CODE GENERATION

The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations? (2024.findings-acl)

Challenge:	Recent advances in large language models (LLMs) have showcased impressive code generation capabilities, primarily evaluated through language-to-code benchmarks.
Approach:	They propose a benchmark to assess LLMs’ code understanding abilities from the perspective of code judging rather than code generation.
Outcome:	The proposed benchmark evaluates 12 well-known large language models to determine the correctness of provided code solutions.

Copied to clipboard

Alex Gu, Wen-Ding Li, Naman Jain, Theo Olausson, Celine Lee, Koushik Sen, Armando Solar-Lezama

Position Paper: How Should We Responsibly Adopt LLMs in the Peer Review Process? (2026.findings-eacl)

Challenge:	Language models are more proficient at code generation, but they still generate incorrect programs.
Approach:	They define a group of models that have a high log-probability and weak correctness checks.
Outcome:	The proposed model samples fail to understand counterfeits through three clear failure modes . counterfeits are confusing to the model as they are to other models, the authors say .

Copied to clipboard

Juhwan Choi, JungMin Yun, Changhun Kim, YoungBin Kim

RAGthoven: A Configurable Toolkit for RAG-enabled LLM Experimentation (2025.coling-demos)

Challenge:	a recent paper criticizes the current use of Large Language Models (LLMs) for simple review text generation.
Approach:	They propose to use Large Language Models to support key aspects of the review process . they argue that this approach overlooks more meaningful applications of LLMs . authors argue that the increased reviewing burden per reviewer is a factor .
Outcome:	The proposed approach would support reproducibility, correctness and relevance of citations and ethics review flagging.

Copied to clipboard

Gregor Karetka, Demetris Skottis, Lucia Dutková, Peter Hraška, Marek Suppa

Challenge:	Large Language Models (LLMs) have significantly altered the landscape of Natural Language Processing (NLP), but their use as a baseline method has not been extensive.
Approach:	They propose a tool for automatic evaluation of RAG-based pipelines that provides a simple yet powerful abstraction.
Outcome:	The proposed tool provides an automatic evaluation of RAG-based pipelines.

How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition (2024.acl-long)

Copied to clipboard

Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, Jingren Zhou

Think Less, Code Better: Probing When Chain-of-Thought Hurts and How to Route Around It (2026.acl-srw)

Challenge:	supervised fine-tuning (SFT) is a technique used to enhance multiple abilities in large language models.
Approach:	They propose to study the interplay of data composition between mathematical reasoning, code generation, and general human-aligning abilities during supervised fine-tuning.
Outcome:	The proposed model improves math reasoning and code generation with increasing data amount . the proposed model size and SFT strategies can be used to learn multiple skills with different scaling patterns.

Copied to clipboard

Rajarshi Ghoshal, Debadri Basak, Salma Emad Mahmoud Abdelhalim, Pratibha Kaur Arora

Tree-of-Evolution: Tree-Structured Instruction Evolution for Code Generation in Large Language Models (2025.acl-long)

Challenge:	Chain-of-Thought (CoT) prompting is the dominant strategy for eliciting step-by-step reasoning in large language models, but its effect on code generation is poorly understood.
Approach:	They develop a chain-of-thought (CoT) prompting router that selects among 12 prompt styles via a single 84 ms forward pass.
Outcome:	The proposed model outperforms CoT in small models with a 84 ms forward pass.

Copied to clipboard

Ziyang Luo, Kaixin Li, Hongzhan Lin, Yuchen Tian, Mohan Kankanhalli, Jing Ma

Improving Repository-level Code Search with Text Conversion (2024.naacl-srw)

Challenge:	Data synthesis is a key research area in large language models (LLMs).
Approach:	They propose a framework that models code instruction synthesis process with a tree structure and optimization-driven evolution to alleviate constraints of unidirectional synthesis and randomness-driven generation.
Outcome:	The proposed framework outperforms open-weight code LLMs on five widely-used benchmarks.

Copied to clipboard

Mizuki Kondo, Daisuke Kawahara, Toshiyuki Kurabayashi

A Study of LLMs’ Preferences for Libraries and Programming Languages (2026.findings-acl)

Challenge:	Existing methods to search for related files based on similarity between code snippets are not effective for repository-level code generation.
Approach:	They propose to take similarities between code snippets and the texts converted from them into LLMs to search for related files and perform generation.
Outcome:	The proposed method improves the accuracy of code search on the repository level.

Copied to clipboard

Lukas Twist, Jie M. Zhang, Mark Harman, Don Syme, Joost Noppen, Helen Yannakoudakis, Detlef Nauck

LETI: Learning to Generate from Textual Interactions (2024.findings-naacl)

Challenge:	Existing evaluations focus on functional correctness or syntactic validity, overlooking how LLMs make critical design choices such as which library or programming language to use.
Approach:	They conduct the first systematic study of LLMs’ preferences for libraries and programming languages when generating code, covering eight different LLM.
Outcome:	The proposed benchmarks show that LLMs prioritize familiarity and popularity over suitability and task-specific optimality.

Copied to clipboard

Xingyao Wang, Hao Peng, Reyhaneh Jabbarvand, Heng Ji

CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimization (2025.findings-emnlp)

Challenge:	Existing techniques fine-tune on input-output pairs or with numerical rewards that gauge the output quality are not effective.
Approach:	They propose to fine-tune pre-trained language models with binary labels and a Python interpreter to get textual feedback from the inputs.
Outcome:	The proposed model outperforms the base model on unseen problems and achieves comparable or better performance on humanEval.

Copied to clipboard

Jing Ye, Rui Wang, Yuchuan Wu, Victor Ma, Feiteng Fang, Fei Huang, Yongbin Li

Dialogue Systems & Conversational Agents

Challenge:	Comparative Policy Optimization (CPO) redefines the reward evaluation paradigm by shifting from sample-wise scoring to comparative group-wise score.
Approach:	They propose a method to optimize subjective tasks by shifting from sample-wise to comparative group-wise scoring.
Outcome:	The proposed framework shifts from sample-wise scoring to comparative group-wise score . it minimizes contextual bias and enables more robust and fair performance evaluation.

MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages (2023.findings-eacl)

Copied to clipboard

Zhiruo Wang, Grace Cuenca, Shuyan Zhou, Frank F. Xu, Graham Neubig

When Language Model Meets Private Library (2022.findings-emnlp)

Challenge:	MCoNaLa benchmarks natural language code generation in languages that are not native to English.
Approach:	They propose to benchmark natural language code generation from natural language commands extending beyond English by using a multilingual dataset.
Outcome:	The proposed dataset compares natural language commands with code generation systems in three languages.

Copied to clipboard

Daoguang Zan, Bei Chen, Zeqi Lin, Bei Guan, Wang Yongji, Jian-Guang Lou

Can docstring reformulation with an LLM improve code generation? (2024.eacl-srw)

Challenge:	Existing language models have been pre-trained on large-scale code corpora and generate decent code snippets.
Approach:	They propose a framework that can provide pre-trained language models with the ability to generate code using private libraries.
Outcome:	The proposed framework can generate code using private libraries using off-the-shelf language models or pre-trained models on code corpus containing API information.

Copied to clipboard

Nicola Dainese, Alexander Ilin, Pekka Marttinen

MiniChain: A Small Library for Coding with Large Language Models (2023.emnlp-demo)

Challenge:	Existing approaches focus on training, fine-tuning or prompting LLMs to generate better outputs given the same input.
Approach:	They propose to optimize part of the input, the docstring, via reformulation with an LLM to improve code generation.
Outcome:	The proposed methods improve code generation on the original HumanEval benchmark and multiple curated variants on the same input.

Copied to clipboard

Alexander Rush

Impeding LLM-assisted Cheating in Introductory Programming Assignments via Adversarial Perturbation (2024.emnlp-main)

Challenge:	Programming augmented by large language models (LLMs) opens up many new application areas, but also requires care.
Approach:	They introduce a tool for augmented programming that provides basic primitives for coding LLM calls.
Outcome:	The proposed tool provides core primitives for coding LLM calls and separating out prompt templates.

Copied to clipboard

Saiful Salim, Rubin Yang, Alexander Cooper, Suryashree Ray, Saumya Debray, Sazzadur Rahaman

A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023.findings-acl)

Challenge:	Large language model (LLM)-based programming assistants can also facilitate cheating in introductory computer science courses.
Approach:	They propose to use Large Language Models to detect and penalize cheating and modify problem statements to impede cheating.
Outcome:	The proposed methods reduce correctness scores by 77% and detectability by perturbations.

Copied to clipboard

Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, Jimmy Huang

Dialogue Systems & Conversational Agents

Challenge:	Currently, the evaluation of large language models (LLMs) such as ChatGPT in academic datasets is difficult due to the difficulty of evaluating the generative outputs produced by this model against the ground truth.
Approach:	They evaluate ChatGPT across 140 tasks and analyze 255K responses it generates in academic datasets.
Outcome:	The proposed model performs well on 140 tasks and generates 255K responses in these datasets.

Reason-Code: Reliable Code Generation via Test-Driven Monte Carlo Tree Search (2026.acl-industry)

Copied to clipboard

Zixu Li, Zhiqi Peng

BanglaSTEM: A Parallel Corpus and Term-Weighted Evaluation for Technical Bangla-English Translation (2026.acl-srw)

Challenge:	Large Language Models (LLMs) are widely used for code generation, but their performance degrades on complex tasks.
Approach:	They propose an inference-time framework that formulates code generation as a search process guided by execution feedback.
Outcome:	The proposed framework improves reliability without paying full cost of additional sampling under strict latency budgets.

Copied to clipboard

Kazi Reyazul Hasan, A. B. M. Alim Al Islam, Muhammad Abdullah Adnan

Multilinguality

Challenge:	Large language models excel at technical problem solving in English but struggle when questions are posed in Bangla.
Approach:	They propose a dataset of 5,000 Bangla-English sentence pairs to align technical terms . they use OCR to extract matching passages from bilingual textbooks .
Outcome:	The proposed pipeline extracts matching passages from bilingual textbooks and uses them to align sentences and mark technical terms.

MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems (2024.findings-emnlp)

Copied to clipboard

Kaixin Li, Yuchen Tian, Qisheng Hu, Ziyang Luo, Zhiyong Huang, Jing Ma

RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation (2024.emnlp-demo)

Challenge:	Programming often involves translating detailed and complex specifications into code . current state-of-the-art models struggle to solve these problems, a new study shows .
Approach:	They propose a multi-modal coding dataset to evaluate algorithmic problem-solving skills in visually rich contexts.
Outcome:	The proposed model lacks powerful vision-code models due to the extreme demand for reasoning abilities.

Copied to clipboard

Qinyu Luo, Yining Ye, Shihao Liang, Zhong Zhang, Yujia Qin, Yaxi Lu, Yesai Wu, Xin Cong, Yankai Lin, Yingli Zhang, Xiaoyin Che, Zhiyuan Liu, Maosong Sun

EvoAgentX: An Automated Framework for Evolving Agentic Workflows (2025.emnlp-demos)

Challenge:	Xia et al., 2018) demonstrate that a large language model can generate and maintain high-quality code documentation.
Approach:	They propose a large language model powered open-source framework for generating, maintaining, and updating code documentation.
Outcome:	The proposed framework generates high-quality documentation for the entire project.

Copied to clipboard

Yingxu Wang, Siwei Liu, Jinyuan Fang, Zaiqiao Meng

Numerical Reasoning

Challenge:	Existing MAS frameworks often require manual workflow configuration and lack native support for dynamic evolution and performance optimization.
Approach:	They propose an open-source platform that automates generation, execution, and evolutionary optimization of multi-agent workflows.
Outcome:	The proposed platform automates generation, execution, and evolutionary optimization of multi-agent workflows.

CodeGuard: Improving LLM Guardrails in CS Education (2026.findings-eacl)

Copied to clipboard

Nishat Raihan, Noah Erdachew, Jayoti Devi, Joanna C. S. Santos, Marcos Zampieri

Rethinking Repetition Problems of LLMs in Code Generation (2025.acl-long)

Challenge:	Large language models (LLMs) are increasingly embedded in Computer Science classrooms to automate code generation, feedback, and assessment.
Approach:	They propose a guardrail framework for educational AI systems that can handle unsafe and irrelevant prompts.
Outcome:	The proposed framework reduces potentially harmful or policy-violating code completions by 30-65% without degrading performance on legitimate educational tasks.

Copied to clipboard

Yihong Dong, Yuchen Liu, Xue Jiang, Bin Gu, Zhi Jin, Ge Li

Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step by Step (2024.findings-acl)

Challenge:	Recent studies have focused on content repetition, but structural repetition is a more prevalent problem in code generation.
Approach:	They propose a decoding approach that eliminates repetition problems in code generation by identifying grammar rules and strategically decaying the likelihood of critical tokens that contribute to repetitions.
Outcome:	The proposed approach outperforms baselines and humanEval benchmarks on CodeRepetEval dataset and MBPP benchmarks, effectively reducing repetitions and enhancing the quality of generated code.

Copied to clipboard

Li Zhong, Zilong Wang, Jingbo Shang

TaCIE: Enhancing Instruction Comprehension in Large Language Models through Task-Centred Instruction Evolution (2025.coling-main)

Challenge:	Large language models (LLMs) are leading progress in code generation, but they are underutilized in the literature.
Approach:	They propose a debugging framework that allows LLMs to refine their generated programs with the runtime execution information.
Outcome:	The proposed framework improves the baseline performance by 9.8% across the HumanEval, MBPP, and TransCoder benchmarks.

Copied to clipboard

Jiuding Yang, Shengyao Lu, Weidong Guo, Xiangyang Li, Kaitong Yang, Yu Xu, Di Niu

Evaluating Tokenizer Adaptation Methods for Large Language Models on Low-Resource Programming Languages (2025.acl-srw)

Challenge:	Existing methods for fine-tuning Large Language Models (LLMs) encounter performance limitations, impeding further enhancements in code generation tasks.
Approach:	They propose to combine two distinct prompts through a hybridization process to enhance the evolution of training prompts for code LLMs.
Outcome:	The proposed method significantly improves the performance of Code LLMs across five code generation benchmarks.

Copied to clipboard

Georgy Andryushchenko, Vladimir V. Ivanov

PLEX: Adaptive Parameter-Efficient Fine-Tuning for Code LLMs using Lottery-Tickets (2025.naacl-industry)

Challenge:	Large language models (LLMs) trained on high-resource programming languages perform sub-optimally for low-resourced programming languages (LRPLs).
Approach:	They evaluate the impact of tokenizer adaptation methods on improving code generation for LRPLs.
Outcome:	The proposed methods outperform the original models and fine-tuned models in LRPLs, but performance declines in non-target languages like Python after tokenizer adaptation.

Copied to clipboard

Jaeseong Lee, Hojae Han, Jongyoon Kim, Seung-won Hwang, Naun Kang, KyungJun An, Sungho Jang

Modular Visual Question Answering via Code Generation (2023.acl-short)

Challenge:	PLEX is a lottery-ticket based parameter-efficient fine-tuning method that adapts large language models to well-supported and underrepresented programming languages (PLs) in pretraining.
Approach:	They propose a lottery-ticket based parameter-efficient fine-tuning method that adapts large language models to well-supported and underrepresented programming languages (PLs)
Outcome:	The proposed method achieves state-of-the-art performance among PEFT methods while maintaining competitive results with reduced computational overhead.

Copied to clipboard

Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, Dan Klein

Testing the Effect of Code Documentation on Large Language Model Code Understanding (2024.findings-naacl)

Challenge:	a framework for visual question answering is based on modular code generation . the scope of reasoning needed for visual questions is vast, and requires many skills .
Approach:	They propose a framework that formulates visual question answering as modular code generation.
Outcome:	The proposed framework improves accuracy on COVR and GQA datasets by 3% and 2% compared to the few-shot baseline that does not employ code generation.

Copied to clipboard

William Macke, Michael Doyle

AMR-Evol: Adaptive Modular Response Evolution Elicits Better Knowledge Distillation for Large Language Models in Code Generation (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have demonstrated impressive abilities in recent years with regards to code generation and understanding.
Approach:	They propose to provide an LLM with "incorrect" documentation that can greatly hinder code understanding, while incomplete or missing documentation does not seem to significantly affect an LRM's ability to understand code.
Outcome:	The proposed model can generate and understand code in a language with high documentation quality while lacking documentation does not significantly affect the ability to understand code.

Copied to clipboard

Ziyang Luo, Xin Li, Hongzhan Lin, Jing Ma, Lidong Bing

E5: Zero-shot Hierarchical Table Analysis using Augmented LLMs via Explain, Extract, Execute, Exhibit and Extrapolate (2024.naacl-long)

Challenge:	proprietary large language models (LLMs) have demonstrated impressive code generation performance.
Approach:	They propose an adaptive module-based model that refines the direct response distillation process by modular decomposition and adaptive response evolution.
Outcome:	The proposed framework outperforms baseline model and code generation methods on three popular benchmarks.

Copied to clipboard

Zhehao Zhang, Yan Gao, Jian-Guang Lou

Exploring Continual Learning for Code Generation Models (2023.acl-short)

Challenge:	Hierarchical tables with multi-level headers are confusing for models due to their complex structure, implicit semantics, and calculation relationships.
Approach:	They propose a code-augmented framework for zero-shot hierarchical table question answering that combines self-explaining the table's hierarchic structures with external code execution to prevent hallucinations.
Outcome:	The proposed framework outperforms state-of-the-art fine-tuning methods with a 44.38 Exact Match improvement.

Copied to clipboard

Prateek Yadav, Qing Sun, Hantian Ding, Xiaopeng Li, Dejiao Zhang, Ming Tan, Parminder Bhatia, Xiaofei Ma, Ramesh Nallapati, Murali Krishna Ramanathan, Mohit Bansal, Bing Xiang

Learning from Few Samples: A Novel Approach for High-Quality Malcode Generation (2025.emnlp-main)

Challenge:	Large-scale code generation models such as Copilot and CodeT5 are expensive to train and re-train.
Approach:	They propose a benchmark for Continual Learning (CL) that covers a wide range of tasks with different input and output programming languages.
Outcome:	The proposed method improves on Prompt Pooling with Teacher Forcing, which suffers catastrophic forgetting due to stark distribution shifts in coding tasks.

Copied to clipboard

Haijian Ma, Daizong Liu, Xiaowen Cai, Pan Zhou, Yulai Xie

Adapting LLMs for Structured Natural Language API Integration (2024.emnlp-industry)

Challenge:	Intrusion detection systems (IDS) are limited in labeled samples due to scarcity and lack of diversity in malicious samples.
Approach:	They propose a semi-supervised framework that integrates Generative Adversarial Networks with Large Language Models to enhance malicious code generation and SQL Injection detection capabilities.
Outcome:	The proposed framework enhances malicious code generation and detection capabilities in few-sample learning scenarios.

Copied to clipboard

Robin Chan, Katsiaryna Mirylenka, Thomas Gschwind, Christoph Miksovic, Paolo Scotton, Enrico Toniato, Abdel Labbi

Named Entity Recognition

Challenge:	API integration is crucial for enterprise systems, but there are challenges in combining APIs based on user intent.
Approach:	They propose a framework that leverages large language models to integrate APIs based on natural language input.
Outcome:	The proposed framework improves performance over existing methods and RAGs based on open APIs . it can learn structural API constraints implicitly during training and retain structured knowledge .

SymCode: A Neurosymbolic Approach to Mathematical Reasoning via Verifiable Code Generation (2026.findings-eacl)

Copied to clipboard

Sina Bagheri Nezhad, Yao Li, Ameeta Agrawal

FROST: Factual Reasoning via Optimized Stochastic Trajectories in Large Language Models during Inference (2026.acl-industry)

Challenge:	Current approaches to mathematical reasoning are inference-time prompting and model fine-tuning.
Approach:	They propose a neurosymbolic framework that reframes mathematical problem-solving as a task of verifiable code generation using the SymPy library.
Outcome:	The proposed framework improves accuracy on MATH-500 and OlympiadBench benchmarks.

Copied to clipboard

Soumedhik Bharati, Ebad Shabbir, Jiechao Gao

Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency (2024.acl-long)

Challenge:	Existing mitigation strategies are needed to improve large language models' reliability and efficiency.
Approach:	They propose an inference-time framework that balances exploration andexploitation without additional training or context augmentation.
Outcome:	FROST achieves 2–5 percentage point improvements over standard chain-of-thoughtprompting and reduces unsupported outputs by 40% relative to Standard CoT.

Copied to clipboard

Baizhou Huang, Shuai Lu, Xiaojun Wan, Nan Duan

On the Impacts of Contexts on Repository-Level Code Generation (2025.findings-naacl)

Challenge:	Existing work utilizes verification properties to verify and re-rank solutions in a majority voting manner, but this assumption may not hold.
Approach:	They propose a multi-perspective self-consistency framework that incorporates both inter- and intra-consistency across outputs from multiple perspectives.
Outcome:	The proposed framework significantly boosts performance of foundation models on various benchmarks, including HumanEval (+15.91%), MBPP (+6.43%) and CodeContests (+9.37%).

Copied to clipboard

Nam Le Hai, Dung Manh Nguyen, Nghi D. Q. Bui

Analyzing the Performance of Large Language Models on Code Summarization (2024.lrec-main)

Challenge:	CodeLLMs are widely used for code generation, but their ability to handle repository-level dependencies remains underexplored.
Approach:	They propose a benchmark for evaluating repository-level code generation based on dependency contexts.
Outcome:	The proposed model improves dependency handling and introduces a new metric, Dependency Invocation Rate (DIR), to measure context utilization.

Copied to clipboard

Rajarshi Haldar, Julia Hockenmaier

UniCoder: Scaling Code Large Language Model via Universal Code (2024.acl-long)

Challenge:	Large language models perform very well on tasks that involve both natural language and source code.
Approach:	They show that large language models perform very well on tasks that involve both natural language and source code.
Outcome:	The proposed models perform very well on tasks that involve both natural language and source code.

Copied to clipboard

Tao Sun, Linzheng Chai, Jian Yang, Yuwei Yin, Hongcheng Guo, Jiaheng Liu, Bing Wang, Liqun Yang, Zhoujun Li

ConCodeEval: Evaluating Large Language Models for Code Constraints in Domain-Specific Languages (2025.acl-industry)

Challenge:	Experimental results show that UniCoder with the universal code significantly outperforms the previous prompting methods by a large margin.
Approach:	They introduce the universal code (UniCode) as the intermediate representation of algorithm steps using conventions of programming languages.
Outcome:	The proposed model outperforms previous prompting methods by a large margin . the proposed model is based on a dataset of natural-language questions and code solutions .

Copied to clipboard

Mehant Kammakomati, Sameer Pimparkhede, Srikanth G. Tamilselvam, Prince Kumar, Pushpak Bhattacharyya

CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions (2026.acl-long)

Challenge:	Large Language Models (LLMs) have demonstrated potential in code generation and natural language understanding, but they struggle with code constraints.
Approach:	They propose to use Large Language Models to handle constraints represented in code . they use JSON, YAML, XML, Python, and natural language to test their effectiveness .
Outcome:	The proposed benchmark shows that LLMs can handle code constraints better than natural language . the results suggest that conscious choice of representations can lead to optimal use of LLM in enterprise use cases involving code constraints.

Copied to clipboard

Jingwei Shi, Xinxiang Yin, Jing Huang, Shengyu Tao, Jinman Zhao

Thesis Proposal: Intentional Inference for Insight Generation (2026.acl-srw)

Challenge:	Existing benchmarks for Large Language Models often lack coverage for subtle corner cases . a substantial amount of effort has been applied to address this challenge .
Approach:	They propose a framework that generates adversarial test cases that expose latent vulnerabilities in code submissions.
Outcome:	The proposed framework improves the True Negative Rate (TNR) of existing datasets and generates superior adversarial cases on liveCodeBench.

Copied to clipboard

Kristýna Onderková

Summarize and Generate to Back-translate: Unsupervised Translation of Programming Languages (2023.eacl-main)

Challenge:	Large language models excel at surface-level fluency but struggle with consistent logical inference beyond surface- level patterns.
Approach:	They propose to shift focus from surface-level generation to assumption-aware deeper inferences . authors argue that underspecification leads to unintentional assumptions .
Outcome:	The proposed model underspecification leads to unintentional assumptions, the authors argue . they also examine how to improve reasoning to enable deeper inferences, focusing on code generation and qualitative reasoning.

Copied to clipboard

Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang

AutoMonitor-Bench: Evaluating the Reliability of LLM-Based Misbehavior Monitor (2026.findings-acl)

Challenge:	Recent developments of multilingual pre-trained sequence-to-sequence models for programming languages have been effective for a broad spectrum of downstream software engineering tasks.
Approach:	They propose to combine a source-to-target model with a target-tosource model trained in parallel.
Outcome:	The proposed approach performs competitively with state-of-the-art methods.

Copied to clipboard

Shu Yang, Jingyu Hu, Tong Li, Hanqi Yan, Wenxuan Wang, Di Wang

Challenge:	AutoMonitor-Bench evaluates the reliability of LLM-based misbehavior monitors across diverse tasks and failure modes.
Approach:	They introduce AutoMonitor-Bench, a benchmark designed to evaluate misbehavior monitors across diverse tasks and failure modes.
Outcome:	The new benchmark evaluates the reliability of LLM-based misbehavior monitors across tasks and failure modes.

INTERVENOR: Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repair (2024.findings-acl)

Copied to clipboard

Hanbin Wang, Zhenghao Liu, Shuo Wang, Ganqu Cui, Ning Ding, Zhiyuan Liu, Ge Yu

Self-Correcting Code Generation Using Small Language Models (2025.findings-emnlp)

Challenge:	Experimental results show that INTERVENOR surpasses baseline models, exhibiting improvements of approximately 18% and 4.3% over GPT-3.5 in code generation and code translation tasks.
Approach:	They propose a system that prompts Large Language Models to play distinct roles during the code repair process, functioning as both a Code Learner and a code teacher.
Outcome:	The proposed system surpasses baseline models in code generation and code translation tasks and improves on syntax errors and assertion errors.

Copied to clipboard

Jeonghun Cho, Deokhyung Kang, Hyounghun Kim, Gary Lee

Can LLMs Learn to Map the World from Local Descriptions? (2026.acl-long)

Challenge:	a recent study has demonstrated that self-correction is a powerful tool for code generation, but whether it is effective for smaller models remains unexplored.
Approach:	They propose a method that trains small language models to maintain correct outputs while progressively correcting incorrect outputs as turns proceed.
Outcome:	The proposed approach improves the ability of small language models for multi-turn code correction.

Copied to clipboard

Sirui Xia, Aili Chen, Xintao Wang, Tinghui Zhu, Yikai Zhang, Jiangjie Chen, Yanghua Xiao

Challenge:	Recent advances in large language models have demonstrated strong capabilities in tasks such as code generation and mathematical reasoning.
Approach:	They investigate whether large language models can construct coherent global spatial cognition by integrating fragmented relational descriptions.
Outcome:	The proposed models can generalize to unseen spatial relationships and exhibit latent representations aligned with real-world spatial distributions.

LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models (2024.emnlp-main)

Copied to clipboard

Yuxuan Wan, Wenxuan Wang, Yiliu Yang, Youliang Yuan, Jen-tse Huang, Pinjia He, Wenxiang Jiao, Michael Lyu

AlphaQT-Bench: Diagnosing the Gap between Financial Code Generation and Quantitative Reasoning in LLMs (2026.findings-acl)

Challenge:	LogicAsker examines and improves the reasoning abilities of large language models such as ChatGPT and GPT-4.
Approach:	They propose a set of atomic reasoning skills grounded in propositional and predicate logic to examine and improve the reasoning abilities of large language models such as ChatGPT and GPT-4.
Outcome:	The proposed approach improves reasoning abilities in large language models such as ChatGPT and GPT-4 by up to 5%.

Copied to clipboard

Sichun Luo, Yi Huang, Shichang Meng, Fengyuan Liu, Mukai Li, Qinghua Yao, Zefa Hu, Junlan Feng, Qi Liu

Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps (2025.findings-acl)

Challenge:	Existing benchmarks rely on outcome-driven metrics such as profitability and look-ahead bias.
Approach:	They propose a diagnostic benchmark for instruction-grounded financial code generation under strict semantic and temporal constraints.
Outcome:	The proposed benchmarks show that the models fail under causal, structural, or functional constraints.

Copied to clipboard

Jiashun Cheng, Aochuan Chen, Nuo Chen, Ziqi Gao, Yuhan Li, Jia Li, Fugee Tsung

EvoR: Evolving Retrieval for Code Generation (2024.findings-emnlp)

Challenge:	Low-Rank Adaptation (LoRA) has emerged as a prominent technique for fine-tuning large foundation models.
Approach:	They propose a low-rank Adaptation technique that harnesses the expressiveness of spectral bases to re-parameterize LoRA from a sparse spectral subspace.
Outcome:	The proposed technique achieves greater efficiency with fewer parameters than baselines on various downstream tasks, including commonsense reasoning, math reasoning, and code generation.

Copied to clipboard

Hongjin Su, Shuyang Jiang, Yuhang Lai, Haoyuan Wu, Boao Shi, Che Liu, Qian Liu, Tao Yu

ICE-Score: Instructing Large Language Models to Evaluate Code (2024.findings-eacl)

Challenge:	Existing pipelines for retrieval-augmented code generation (RACG) use static knowledge bases with a single source, limiting adaptation capabilities of Large Language Models (LLMs) Extensive experiments demonstrate that EVOR achieves two to four times of execution accuracy compared to other methods such as Reflexion.
Approach:	They propose a retrieval-augmented code generation pipeline that employs the synchronous evolution of queries and diverse knowledge bases.
Outcome:	The proposed pipeline achieves two to four times of execution accuracy compared to other methods.

Copied to clipboard

Terry Yue Zhuo

PairCoder: Pair Programming-Inspired Two-Agent Collaboration for Code Generation (2026.findings-acl)

Challenge:	Recent advances in the field of natural language generation have facilitated the use of large language models to assess the quality of generated text.
Approach:	They propose a new evaluation metric by instructing large language models for code assessments using a set of programming languages.
Outcome:	The proposed evaluation metric surpasses state-of-the-art metrics for code generation, delivering high levels of accuracy and consistency across programming languages and tasks.

Copied to clipboard

Junhao Chen, Xiang Li, Yibin Xu, Yuehan Cui, Fangsheng Weng, Hao Zhao, Fei Ma, Qi Tian

UIOrchestra: Generating High-Fidelity Code from UI Designs with a Multi-agent System (2025.findings-emnlp)

Challenge:	Existing multi agent frameworks for large language models are brittle on code generation tasks.
Approach:	They propose a framework that brings pair programming to autonomous LLM collaboration.
Outcome:	Using PairCoder, large language models achieve better results on code generation tasks and reduce token usage by 40% to 70% on eight representative backbones.

Copied to clipboard

Chuhuai Yue, Jiajun Chai, Yufei Zhang, Zixiang Ding, Xihao Liang, Peixin Wang, Shihai Chen, Wang Yixuan, null Wangyanping, Guojun Yin, Wei Lin

ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use (2025.acl-long)

Challenge:	Recent advances in large language models have significantly improved automated code generation . however, the translation of complex mobile UI designs into high-fidelity front-end code remains a challenge .
Approach:	They propose a collaborative multi-agent system to reconstruct static single-page apps from mockups.
Outcome:	The proposed system outperforms existing methods in reconstructing complex app pages . the code and data will be released upon paper acceptance .

Copied to clipboard

Junjie Ye, Zhengyin Du, Xuesong Yao, Weijian Lin, Yufei Xu, Zehui Chen, Zaiyuan Wang, Sining Zhu, Zhiheng Xi, Siyu Yuan, Tao Gui, Qi Zhang, Xuanjing Huang, Jiecao Chen

Benchmarking Testing in Automated Theorem Proving (2026.acl-industry)

Challenge:	Effective evaluation of multi-hop tool use is critical for analyzing the understanding, reasoning, and function-calling capabilities of large language models.
Approach:	They propose a dataset that provides rigorous evaluation of multi-hop tool use.
Outcome:	The proposed model achieves 49.04% accuracy across five model families.

Copied to clipboard

Jongyoon Kim, Hojae Han, Seung-Won Hwang

Challenge:	Existing evaluations rely on indirect proxies such as lexical overlap with human-annotated proof, or expensive manual inspection.
Approach:	They propose a framework that evaluates the semantic correctness of formal theorems . they use a set of problems paired with 41 successor theorels to compare them .
Outcome:	The proposed framework evaluates the semantic correctness of formal theorems using real-world Lean 4 repositories.

VisCoder: Fine-Tuning LLMs for Executable Python Visualization Code Generation (2025.findings-emnlp)

Copied to clipboard

Yuansheng Ni, Ping Nie, Kai Zou, Xiang Yue, Wenhu Chen

Evaluating In-Context Learning of Libraries for Code Generation (2024.naacl-long)

Challenge:	Existing instruction-tuning datasets lack execution-grounded supervision and offer limited support for iterative code correction.
Approach:	They propose a large-scale instruction tuning dataset for Python-based visualization and self-correction.
Outcome:	The proposed dataset outperforms strong open-source baselines and proprietary models like GPT-4o-mini.

Copied to clipboard

Arkil Patel, Siva Reddy, Dzmitry Bahdanau, Pradeep Dasigi

CODERL+: Improving Code Generation via Reinforcement with Execution Semantics Alignment (2026.acl-long)

Challenge:	Recent work shows that large proprietary LLMs can learn novel library usage in-context from demonstrations.
Approach:	They evaluate large proprietary LLMs to understand library usage in-context . they find they are able to generate code based on library specification presented in-constext - a promising area .
Outcome:	The proposed models can learn library usage in-context from demonstrations . the results pave the way for more adaptable and dynamic coding environments.

Copied to clipboard

Xue Jiang, Yihong Dong, Mengyang Liu, Deng Hongyi, Tian Wang, Yongding Tao, Zhi Jin, Wenpin Jiao, Ge Li

Saber: Efficient Sampling with Adaptive Acceleration and Backtracking Enhanced Remasking for Diffusion Language Model in Code Generation (2026.acl-long)

Challenge:	Large Language Models excel at code generation by learning from vast code corpora, but a fundamental semantic gap remains between training on textual patterns and the goal of functional correctness . reinforcement learning with verifiable rewards (RLVR) approaches are inefficient for establishing a well-aligned connection between the textual representation of code and its execution semantics.
Approach:	They propose a novel approach that integrates execution semantics alignment into the RLVR training pipeline for code generation.
Outcome:	The proposed model outperforms baseline training and RLVR and shows strong applicability across RL and LLMs.

Copied to clipboard

Yihong Dong, Zhaoyu Ma, Xue Jiang, Zhiyuan Fan, Jiaru Qian, Yongmin Li, Jianha Xiao, Zhi Jin, Ge Li

AdaptFlow: Adaptive Workflow Optimization via Meta-Learning (2025.findings-emnlp)

Challenge:	Diffusion language models (DLMs) offer advantages in parallel generation and bidirectional context modeling, but they face a critical trade-off between inference speed and output quality for tasks with strict structural constraints such as code generation.
Approach:	They propose an efficient sampling algorithm that reduces the number of tokens unmasked per step based on the model’s evolving confidence.
Outcome:	The proposed method improves Pass@1 accuracy by 1.9% while achieving 251.4% inference speedup.

Copied to clipboard

Runchuan Zhu, Bowen Jiang, Lingrui Mei, Fangkai Yang, Lu Wang, Haoxiang Gao, Fengshuo Bai, Pu Zhao, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

CodeRAG-Bench: Can Retrieval Augment Code Generation? (2025.findings-naacl)

Challenge:	Existing approaches to large language models rely on static templates or manual workflows.
Approach:	AdaptFlow is a language-based meta-learning framework inspired by model-agnostic meta- learning.
Outcome:	AdaptFlow outperforms manual and automated workflows on question answering, code generation and mathematical reasoning benchmarks.

Copied to clipboard

Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F. Xu, Yiqing Xie, Graham Neubig, Daniel Fried

Quality Assessment of Tabular Data using Large Language Models and Code Generation (2025.emnlp-industry)

Challenge:	Language models excel at generating code, but many programs are difficult to generate using only parametric knowledge.
Approach:	They propose a retrieval-augmented code generation benchmark that provides reproducible evaluations on retrieval and end-to-end code generation performance.
Outcome:	The proposed benchmark covers programming, open-domain, and repository-level tasks and provides reproducible evaluations on retrieval and end-to-end code generation performance.

Copied to clipboard

Ashlesha Akella, Akshar Kaul, Krishnasuri Narayanam, Sameep Mehta

Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering (2025.naacl-long)

Challenge:	Data quality is vital for business decisions; poor data quality costs organizations an average of $12.9 million annually.
Approach:	They propose a framework that combines statistical inliner detection with LLM-driven rule and code generation.
Outcome:	The proposed framework produces semantically valid quality rules and validates them with retrieval-augmented generation (RAG) Extensive evaluations on benchmark datasets confirm the effectiveness of the proposed framework.

Copied to clipboard

Chenglei Si, Yanzhe Zhang, Ryan Li, Zhengyuan Yang, Ruibo Liu, Diyi Yang

Jointly Learning Semantic Parser and Natural Language Generator via Dual Information Maximization (P19-1)

Challenge:	Generative AI has made rapid advances in multimodal understanding and code generation.
Approach:	They construct a first real-world benchmark for multimodal large language models that directly convert visual designs into code implementations by manually curating 484 diverse real-life webpages as test cases.
Outcome:	The proposed model can generate code implementations that directly render into the given reference webpages, given the screenshots as input.

Copied to clipboard

Hai Ye, Wenjie Li, Lu Wang

Semantic Parsing

Challenge:	Semantic parsing aims to transform natural language utterances into formal meaning representations (MRs) whereas an NL generator achieves the reverse, the two tasks are often studied separately.
Approach:	They propose a method of dual information maximization to regularize the learning process by matching the joint distributions of p and q of NLs.
Outcome:	The proposed method empirically maximizes the variational lower bounds of expected joint distributions of NL and MRs.

Unified Pre-training for Program Understanding and Generation (2021.naacl-main)

Copied to clipboard

Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang

Instruction Fusion: Advancing Prompt Evolution through Hybridization (2024.acl-long)

Challenge:	PLUG is a programming language that is used for programming and language understanding and generation tasks.
Approach:	They propose a sequence-to-sequence model that performs a broad spectrum of program and language understanding and generation tasks.
Outcome:	The proposed model outperforms or rivals state-of-the-art models on code summarization, code generation, and code translation tasks in seven programming languages.

Copied to clipboard

Weidong Guo, Jiuding Yang, Kaitong Yang, Xiangyang Li, Zhuwei Rao, Yu Xu, Di Niu

SolEval: Benchmarking Large Language Models for Repository-level Solidity Smart Contract Generation (2025.emnlp-main)

Challenge:	Existing methods for fine-tuning Large Language Models (LLMs) encounter performance limitations, impeding further enhancements in code generation tasks.
Approach:	They propose to combine two distinct prompts through a hybridization process to enhance the evolution of training prompts for code LLMs.
Outcome:	The proposed method significantly improves the performance of Code LLMs across five code generation benchmarks, namely HumanEval, HumanEva+, MBPP, mbap+ and MultiPL-E.

Copied to clipboard

Zhiyuan Peng, Xin Yin, Rui Qian, Peiqin Lin, YongKang Liu, Hao Zhang, Chenhao Ying, Yuan Luo

Functional Overlap Reranking for Neural Code Generation (2024.findings-acl)

Challenge:	Existing methods focus on Python and Java, neglecting Solidity, the programming language for Ethereum smart contracts.
Approach:	They construct a repository-level benchmark for Solidity to evaluate the performance of LLMs on Ethereum.
Outcome:	The proposed benchmarks show that the best performing LLM achieves only 26.29% Pass@10, highlighting room for improvement in Solidity code generation.

Copied to clipboard

Hung To, Minh Nguyen, Nghi Bui

FrontCoder: Scaling Visual Fidelity in Front-End Code Generation (2026.findings-acl)

Challenge:	Code Large Language Models (CodeLLMs) have ushered in a new era in code generation, but selecting the best code solutions remains a challenge.
Approach:	They propose a new reranking strategy that quantifies the functional overlap between solution clusters to provide a better ranking strategy for code solutions.
Outcome:	Empirical results show that the proposed method surpasses state-of-the-art methods on the pass@1 score.

Copied to clipboard

Jun Feng, Jian Yang, Wei Zhang, Jing Wang, Keyi Chen, Xiaokun Yang, Weicheng Gu, Yihang Lou, Yan Bai, Xianglong Liu

Building Language Models for Text with Named Entities (P18-1)

Challenge:	Existing work on front-end code generation fails to provide visual fidelity and rendering quality for front- end developers.
Approach:	They propose a three-stage pipeline to enhance front-end code generation capabilities in LLMs . they use synthetic data, quality-controlled supervised fine-tuning, and reinforcement learning .
Outcome:	The proposed model achieves competitive performance with frontier models while maintaining generation efficiency.

Copied to clipboard

Md Rizwan Parvez, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang

RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesis (2025.emnlp-main)

Challenge:	Existing language models fail to predict the entity names due to their wide variations.
Approach:	They propose a language model which can learn the entity names by leveraging their entity type information.
Outcome:	The proposed model achieves 52.2% better perplexity in recipe generation and 22.06% on code generation than state-of-the-art language models.

Copied to clipboard

Jianwei Wang, Chengming Shi, Junyao Yang, Haoran Li, Qianli Ma, Huiping Zhuang, Cen Chen, Ziqian Zeng

Retrieval Augmented Code Generation and Summarization (2021.findings-emnlp)

Challenge:	Existing solutions to fine-tune large language models for domain-specific tasks are ineffective in addressing privacy concerns.
Approach:	They propose a privacy-preserving framework that fine-tunes a reward proxy model and uses reward signals to guide the synthetic data generation.
Outcome:	The proposed framework fine-tunes a reward proxy model and uses reward signals to guide the synthetic data generation.

Copied to clipboard

Md Rizwan Parvez, Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang

P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs (2025.emnlp-main)

Challenge:	Software developers often recall parts of source code or code summaries that they had written in the past while implementing software or documenting them.
Approach:	They propose a retrieval augmented framework that retrieves relevant code or summaries from a database and provides them as a supplement to code generation or summarization models.
Outcome:	The proposed framework can search for relevant code or summaries from retrieval databases and can work with unimodal (only code or natural language description) or bimodal instances (code-description pairs).

Copied to clipboard

Yidan Zhang, Yu Wan, Boyi Deng, Baosong Yang, Hao-Ran Wei, Fei Huang, Bowen Yu, Dayiheng Liu, Junyang Lin, Fei Huang, Jingren Zhou

Multilinguality

Challenge:	Recent advances in large language models showcase varied multilingual capabilities across tasks . previous assessments focused on fundamental natural language processing (NLP) or isolated capability-specific tasks.
Approach:	They propose a multilingual multitask benchmark to assess multilingual capabilities . they use a large-scale benchmark covering fundamental and capability-specialized datasets .
Outcome:	The proposed benchmark compares models and tasks across languages and tasks and examines knowledge transfer from English to other languages.

DebugBench: Evaluating Debugging Capability of Large Language Models (2024.findings-acl)

Copied to clipboard

Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Hui Haotian, Liu Weichuan, Zhiyuan Liu, Maosong Sun

StepCoder: Improving Code Generation with Reinforcement Learning from Compiler Feedback (2024.acl-long)

Challenge:	Large language models (LLMs) have demonstrated exceptional coding capabilities, but their debugging capabilities remain relatively unexplored.
Approach:	They propose a debugging benchmark consisting of 4,253 LLMs with four major bug categories and 18 minor types in C++, Java, and Python.
Outcome:	The proposed benchmark covers four major bug categories and 18 minor types in C++, Java, and Python.

Copied to clipboard

Shihan Dou, Yan Liu, Haoxiang Jia, Enyu Zhou, Limao Xiong, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, Zhiheng Xi, Yuhao Zhou, Tao Ji, Rui Zheng, Qi Zhang, Tao Gui, Xuanjing Huang

DolphCoder: Echo-Locating Code Large Language Models with Diverse and Multi-Objective Instruction Tuning (2024.acl-long)

Challenge:	Existing work integrates reinforcement learning with compiler feedback to enhance code generation quality but the long code generated by LLMs makes RL exploration ineffective.
Approach:	They propose a framework that integrates reinforcement learning and compiler feedback to enhance code generation quality.
Outcome:	The proposed framework outperforms state-of-the-art approaches in corresponding benchmarks and integrates reinforcement learning with compiler feedback to improve code generation quality.

Copied to clipboard

Yejie Wang, Keqing He, Guanting Dong, Pei Wang, Weihao Zeng, Muxi Diao, Weiran Xu, Jingang Wang, Mengdi Zhang, Xunliang Cai

Aligning Complex Knowledge Graph Question Answering as Knowledge-Aware Constrained Code Generation (2025.coling-main)

Challenge:	Numerous code large language models (LLMs) have been proposed to enhance code generation performance.
Approach:	They propose a diverse instruction model DolphCoder with self-evaluating for code generation that learns diverse instruction targets and combines a code evaluation objective to enhance its code generation ability.
Outcome:	The proposed model achieves superior performance on the HumanEval and MBPP benchmarks, demonstrating new insights for future code instruction tuning work.

Copied to clipboard

Prerna Agarwal, Nishant Kumar, Srikanta Bedathur Jagannath

Krikri: Advancing Open Large Language Models for Greek (2025.findings-emnlp)

Challenge:	Existing frameworks that generate LF using Large Language Models (LLMs) in a few-shot setting are limited due to little exposure to the LF during pre-training.
Approach:	They propose a framework that aligns the LF generation as code generation that incorporates LF-specific constraints.
Outcome:	The proposed framework surpasses all few-shot baselines on KQA Pro by 21%.

Copied to clipboard

Dimitris Roussis, Leon Voukoutis, Georgios Paraskevopoulos, Sokratis Sofianopoulos, Prokopis Prokopidis, Vassilis Papavassileiou, Athanasios Katsamanis, Stelios Piperidis, Vassilis Katsouros

MapCoder: Multi-Agent Code Generation for Competitive Problem Solving (2024.acl-long)

Challenge:	Llama-Krikri-8B is a cutting-edge Large Language Model for the Greek language based on Meta's Llma 3.1-8B.
Approach:	They propose to use Llama-Krikri-8B to train Greek language models . it has 8 billion parameters and is capable of handling polytonic text and Ancient Greek .
Outcome:	The proposed model is based on Meta's Llama 3.1-8B and has 8 billion parameters and is capable of handling polytonic text and Ancient Greek.

Copied to clipboard

Md. Ashraful Islam, Mohammed Eunus Ali, Md Rizwan Parvez

Duplicate-Aware Controlled Code Generation: Enhancing Copyright Protection with Targeted Reordering Beam Search in LLMs (2026.findings-acl)

Challenge:	Large language models (LLMs) have impressive proficiency in natural language processing, but performance in code generation tasks remains limited.
Approach:	They propose a framework that emulates the full cycle of program synthesis as observed in humans.
Outcome:	The proposed framework replicates the full cycle of program synthesis as observed in human developers.

Copied to clipboard

Junbo Fu, Guoshuai Zhao, Linkang Yang, Yunqi Mi, Xueming Qian

Beyond the Surface: A Solution-Aware Retrieval Model for Competition-level Code Generation (2025.findings-emnlp)

Challenge:	Experimental results demonstrate that TRBS effectively reduces verbatim repetition while maintaining functional adequacy.
Approach:	They propose a plug-and-play decoding method that dynamically reorders beam candidates to reduce direct copying.
Outcome:	The proposed method reduces verbatim repetition while maintaining functional adequacy on a multi-language code generation benchmark.

Copied to clipboard

Shiwen Zhang, Lingxiang Wang, Hainan Zhang, Ziwei Wang, Sijia Wen, Zhiming Zheng

Insertion-based Tree Decoding (2021.findings-acl)

Challenge:	Existing retrieval models emphasize surface-level semantic similarity, neglecting deeper solution-level logical similarities.
Approach:	They propose a solution-aware ranking model empowered by synthetic data for competitive programming tasks.
Outcome:	The proposed ranking model outperforms existing retrieval models in precision and recall metrics.

Copied to clipboard

Denis Lukovnikov, Asja Fischer

Unlocking the Potential of Diffusion Language Models through Template Infilling (2026.acl-long)

Challenge:	Existing non-autoregressive decoders that are sub-linear can speed up inference for longer sequences.
Approach:	They propose a sub-linear nonautoregressive tree decoder that uses tree-based insertion operations to generate trees in sub-lines . they evaluate their approach on semantic parsing and compare it against strong baselines .
Outcome:	The proposed approach achieves competitive accuracies while reducing the number of decoding steps.

Copied to clipboard

Junhoo Lee, Seungyeon Kim, Nojun Kwak

CodeSim: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debugging (2025.findings-naacl)

Challenge:	Existing methods rely on prefix-based prompting, resulting in a lack of stability and a large computational time.
Approach:	They propose a conditioning methodology tailored for Diffusion Language Models that distributes structural anchors across the target response, establishing a global template before infilling masked segments.
Outcome:	The proposed method improves on mathematical reasoning, code generation, and trip planning benchmarks while maintaining speed and robustness.

Copied to clipboard

Md. Ashraful Islam, Mohammed Eunus Ali, Md Rizwan Parvez

ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Models (2026.findings-acl)

Challenge:	Large Language Models (LLMs) have made significant strides in code generation and problem solving.
Approach:	They propose a multi-agent code generation framework that integrates human-like perception to address the stages of program synthesis.
Outcome:	The proposed framework achieves state-of-the-art (pass@1) results and shows potential for even greater enhancement when cascaded with external debuggers.

Copied to clipboard

Zhangyue Yin, Qiushi Sun, Zhiyuan Zeng, Zhiyuan Yu, Qipeng Guo, Xuanjing Huang, Xipeng Qiu

CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2024.acl-long)

Challenge:	Existing evaluation methods for test-time scaling are limited.
Approach:	They propose an adaptive resolution-aware scaling evaluation metric specifically designed to assess the test-time scaling effectiveness of large reasoning models.
Outcome:	The proposed metric provides a reliable and fine-grained measurement of test-time scaling capabilities, revealing significant variations in scaling efficiency across models.

Copied to clipboard

Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Hari Sundaram, Shuiguang Deng

SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding (2024.acl-long)

Challenge:	Existing benchmarks for evaluating the code understanding and generation capacities of Large Language Models are insufficient . existing benchmarks focus on a narrow range of popular programming languages and specific tasks .
Approach:	They propose an execution-based, multilingual, multitask evaluation benchmark for LLMs . they evaluate coding performance from three dimensions: length, difficulty, efficiency .
Outcome:	The proposed benchmark covers 43 programming languages and eight coding tasks.

Copied to clipboard

Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, Radha Poovendran

Beyond Sequences: Two-dimensional Representation and Dependency Encoding for Code Generation (2025.acl-long)

Challenge:	Despite advances in large language models, they face substantial challenges in terms of safety.
Approach:	They develop a safety-aware decoding strategy for large language models to defend against jailbreak attacks.
Outcome:	The proposed strategy outperforms six defense methods against jailbreak attacks on five LLMs.

Copied to clipboard

Xiangyu Zhang, Yu Zhou, Guang Yang, Wei Cheng, Taolue Chen

The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation (2023.findings-emnlp)

Challenge:	Existing code generation approaches represent code as a linear sequence of tokens, but positional encodings compromise generalization . explicit positional encoders sacrifice permutation invariance, imposes a strict order on the input sequence .
Approach:	They propose to represent code snippets as two-dimensional entities with explicit encodings . they propose to use dictionary learning to perform semantic matching between code lines .
Outcome:	The proposed model captures the hierarchical and spatial structure of code, especially the dependencies between code lines.

Copied to clipboard

Dung Nguyen, Le Nam, Anh Dau, Anh Nguyen, Khanh Nghiem, Jin Guo, Nghi Bui

SEK: Self-Explained Keywords Empower Large Language Models for Code Generation (2025.findings-acl)

Challenge:	Open-source dataset of code-text pairs for training large language models to understand code is outperforms other datasets for code generation and understanding tasks.
Approach:	They propose to extract high-quality code-text pairs from a dataset of 43 million pairs . they use rules and deep learning to ensure that the code-sampled samples contain high-quality pairs a .
Outcome:	The Vault dataset outperforms existing models on common coding tasks . authors hope the results will propel AI research and software development forward .

Copied to clipboard

Lishui Fan, Mouxiang Chen, Zhongxin Liu

DeepRTL2: A Versatile Model for RTL-Related Tasks (2025.findings-acl)

Challenge:	Large language models (LLMs) have achieved impressive performance in code generation.
Approach:	They propose a technique that extracts and explicates the key terms in the problem description with the LLM itself.
Outcome:	The proposed technique improves the Pass@1 of DeepSeek-Coder-V2-Instruct from 85.4% to 93.3% on the humaneval benchmark.

Copied to clipboard

Yi Liu, Hongji Zhang, Yunhao Zhou, Zhengyuan Shi, Changran Xu, Qiang Xu

RMoA: Optimizing Mixture-of-Agents through Diversity Maximization and Residual Compensation (2025.findings-acl)

Challenge:	Integration of large language models into electronic design automation has been a key driver in eDA.
Approach:	They propose a family of large language models that unifies generation- and embedding-based tasks related to RTL.
Outcome:	The proposed model achieves state-of-the-art performance across all evaluated tasks.

Copied to clipboard

Zhentao Xie, Chengcheng Han, Jinxin Shi, Wenjun Cui, Xin Zhao, Xingjiao Wu, Jiabao Zhao

Boundary-Guided Policy Optimization for Memory-efficient RL of Diffusion Large Language Models (2026.acl-long)

Challenge:	Multi-agent systems based on large language models are limited by high computational overhead, information loss, and robustness.
Approach:	They propose a Residual Mixture-of-Agents (RMoA) that integrates residual connections to optimize efficiency and reliability.
Outcome:	The proposed model achieves state-of-the-art performance on benchmarks of alignment, mathematical reasoning, code generation, and multitasking understanding, while significantly reducing computational overhead.

Copied to clipboard

Nianyi Lin, Jiajie Zhang, Lei Hou, Juanzi Li

Dynamic Scaling of Unit Tests for Code Reward Modeling (2025.acl-long)

Challenge:	Existing methods for reinforcement learning (RL) require a large sample size to be implemented.
Approach:	They propose a memory-efficient RL algorithm that maximizes a lower bound of the ELBO-based objective.
Outcome:	Experiments show that BGPO outperforms previous RL algorithms for diffusion large language models in math problem solving, code generation, and planning tasks.

Copied to clipboard

Zeyao Ma, Xiaokang Zhang, Jing Zhang, Jifan Yu, Sijia Luo, Jie Tang

CodeSCM: Causal Analysis for Multi-Modal Code Generation (2025.naacl-long)

Challenge:	Existing large language models struggle to produce accurate responses on the first attempt for complex reasoning tasks like code generation.
Approach:	They propose a lightweight yet effective unit test generator that scales unit tests based on problem difficulty.
Outcome:	The proposed approach significantly improves performance on three benchmarks.

Copied to clipboard

Mukur Gupta, Noopur Bhatt, Suman Jana

CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflow (2024.findings-acl)

Challenge:	Prior work has shown that multimodal prompts can be highly sensitive, where small adjustments might result in drastically different responses from the model.
Approach:	They propose a Structural Causal Model (SCM) for analyzing multi-modal code generation using large language models (LLMs).
Outcome:	The proposed model is based on the principles of Causal Mediation Analysis and quantifies the causal effects of different prompt modalities on the model.

Copied to clipboard

Nathanaël Beau, Benoit Crabbé

PROMTEC: Fast LLM Inference Decoding using Prompt Multi-Lookup with Template Database and Common Sequences (2025.findings-acl)

Challenge:	Comprising 3,402 crafted examples, our dataset is designed for both model finetuning and standalone evaluation.
Approach:	They propose a dataset that provides examples that include a clarified intent, code snippets associated, and an average of three related unit tests.
Outcome:	The proposed dataset includes 3,402 hand-written examples and 3,121 unrefined examples.

Copied to clipboard

Alan Chi-Man Lee, Wing-Sun Cheng, Calvin Chun-Kit Chan

Low-Resource NLP

Challenge:	Prompt Multi-Lookup, Template Datastore, and Common Sequences methods are used to accelerate the inference of large language models (LLMs).
Approach:	They propose a multi-faceted approach to accelerate the inference of large language models by leveraging Prompt Multi-Lookup, Template Datastore, and Common Sequences methods.
Outcome:	The proposed method achieves a 3.91 speedup on the miniF2F benchmark and 4.23 on the HumanEval benchmark.

DependEval: Benchmarking LLMs for Repository Dependency Understanding (2025.findings-acl)

Copied to clipboard

Junjia Du, Yadi Liu, Hongcheng Guo, Jiawei Wang, Haojian Huang, Yunyi Ni, Zhoujun Li

Challenge:	a benchmark is designed to evaluate the repository-level dependency understanding of large language models (LLMs) based on 2683 repositories from real-world websites.
Approach:	They propose a benchmark to evaluate repository dependency understanding for large language models . DEPENDEVAL evaluates models on three core tasks across 8 programming languages .
Outcome:	The benchmark evaluates models on three core tasks across 8 programming languages from real-world repositories.

Turning the Tide: Repository-based Code Reflection (2025.findings-emnlp)

Copied to clipboard

Wei Zhang, Jian Yang, Jiaxi Yang, Ya Wang, Zhoujun Li, Zeyu Cui, Binyuan Hui, Junyang Lin

Analysis of Tree-Structured Architectures for Code Generation (2021.findings-acl)

Challenge:	Code large language models (LLMs) enhance programming by understanding and generating code across languages.
Approach:	a new benchmark evaluates code understanding and generation in repositories using code large language models.
Outcome:	The proposed model improves code understanding and generation in repositories by evaluating 1,888 test cases across 6 programming languages.

Copied to clipboard

Samip Dahal, Adyasha Maharana, Mohit Bansal

Exploring Dynamic Selection of Branch Expansion Orders for Code Generation (2021.acl-long)

Challenge:	Code generation is the task of generating code snippets from input user specifications written in natural language (NL).
Approach:	They evaluate the significance of input parse trees for code generation by using constituency-based parsers as input and an abstract syntax tree as the target.
Outcome:	The proposed models on a Python-based code generation dataset and a semantic parsing dataset show that constituency trees encoded using a structure-aware model improve performance.

Copied to clipboard

Hui Jiang, Chulun Zhou, Fandong Meng, Biao Zhang, Jie Zhou, Degen Huang, Qingqiang Wu, Jinsong Su

Decoding at the Speed of Thought: Harnessing Parallel Decoding of Lexical Units for LLMs (2024.lrec-main)

Challenge:	Existing code generation models model abstract syntax tree (AST) but not suitable for all multi-branch nodes.
Approach:	They propose to equip a Seq2Tree model with a branch selector to determine optimal expansion orders for multi-branch nodes.
Outcome:	The proposed model can determine optimal expansion orders of branches for multi-branch nodes.

Copied to clipboard

Chenxi Sun, Hongzhi Zhang, Zijia Lin, Jingyuan Zhang, Fuzheng Zhang, Zhongyuan Wang, Bin Chen, Chengru Song, Di Zhang, Kun Gai, Deyi Xiong

Scaling LLM Inference Efficiently with Optimized Sample Compute Allocation (2025.naacl-long)

Challenge:	Large language models have demonstrated exceptional capability in natural language understanding and generation, but their generation speed is limited by the inherently sequential nature of their decoding process.
Approach:	They propose a method that accelerates decoding process without sacrificing quality . they propose lexical unit decoding, which can be integrated with other methods .
Outcome:	The proposed method significantly reduces decoding time while maintaining quality while maintaining output quality.

Copied to clipboard

Kexun Zhang, Shang Zhou, Danqing Wang, William Yang Wang, Lei Li

RethinkMCTS: Refining Erroneous Thoughts in Monte Carlo Tree Search for Code Generation (2025.emnlp-main)

Challenge:	Existing methods to optimize sample allocations for large language models fail to account for the optimal sampling configuration.
Approach:	They propose an algorithm that optimizes sample allocation by finding an optimal mix of different inference configurations.
Outcome:	The proposed algorithm achieves better accuracy on SWE-Bench with 3x less compute than the default configuration.

Copied to clipboard

Qingyao Li, Wei Xia, Xinyi Dai, Kounianhua Du, Weiwen Liu, Yasheng Wang, Ruiming Tang, Yong Yu, Weinan Zhang

RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository (2026.acl-long)

Challenge:	Existing tree search methods neglect the underlying reasoning process, resulting in poor search quality.
Approach:	They propose a framework that systematically explores and refines the reasoning process for code generation by using a tree search engine and a reflection mechanism.
Outcome:	The proposed framework outperforms existing methods in the code generation domain.

Copied to clipboard

Zhiyuan Peng, Xin Yin, Pu Zhao, Fangkai Yang, Lu Wang, Ran Jia, Xu Chen, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

Challenge:	Existing benchmarks focus on isolated function/class-level generation, neglecting complete microservice repository generation.
Approach:	They propose a multilingual benchmark for repository-level end-to-end web microservice generation that reflects real-world development workflows.
Outcome:	The benchmark compared 106 repositories across 18 domains and 11 frameworks and 1,258 API endpoints and 2,335 test cases.

Large Language Models for Predictive Analysis: How Far Are They? (2025.findings-acl)

Copied to clipboard

Qin Chen, Yuanyi Ren, Xiaojun Ma, Yuyang Shi

Personalized Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generation (2023.emnlp-main)

Challenge:	Existing studies on LLMs do not evaluate their capability in predictive analysis.
Approach:	They propose a benchmark to evaluate Large Language Models (LLMs) they integrate 1130 queries from 44 real-world datasets of 8 different fields to evaluate their capability .
Outcome:	The proposed benchmark evaluates 12 renowned LLMs from 44 real-world datasets . results offer insights into their practical use in predictive analysis .

Copied to clipboard

Hailin Chen, Amrita Saha, Steven Hoi, Shafiq Joty

Enhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mapping (2025.findings-acl)

Challenge:	Recent studies have shown that close-sourced LLMs lack the ability to integrate into real-world applications due to their high associated costs and ethical concerns.
Approach:	They propose to use student model to refine its own solution by querying ChatGPT to generate task instruction and solution pairs and querying data to refine model.
Outcome:	The proposed model outperforms standard distillation with only one third of the data.

Copied to clipboard

Yijie Chen, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, Jie Zhou

Flow2Code: Evaluating Large Language Models for Flowchart-based Code Generation Capability (2025.findings-acl)

Challenge:	Knowledge distillation (KD) approaches focus on homogeneous architectures with identical tokenizers, constraining their applicability in cross-architecture scenarios.
Approach:	They propose a framework that uses contextual information to enhance sequence alignment precision and dynamically improves vocabulary mapping.
Outcome:	The proposed framework shows significant advantages over existing methods for model compression . it can be used across multiple model families and across multiple benchmarks .

Copied to clipboard

Mengliang He, Jiayi Zeng, Yankai Jiang, Wei Zhang, Zeming Liu, Xiaoming Shi, Aimin Zhou

CodePRM: Execution Feedback-enhanced Process Reward Model for Code Generation (2025.findings-acl)

Challenge:	Existing code generation benchmarks neglect flowchart-based code generation . existing benchmarks lack flowcharting-based evaluation, limiting the potential of large language models and minimizing human error.
Approach:	They propose to use flowcharts to evaluate existing LLMs' code generation capabilities.
Outcome:	The proposed benchmarks show that the supervised fine-tuning technique contributes greatly to the models’ performance.

Copied to clipboard

Qingyao Li, Xinyi Dai, Xiangyang Li, Weinan Zhang, Yasheng Wang, Ruiming Tang, Yong Yu

CGBridge: Bridging Code Graphs and Large Language Models for Better Structure-Aware Code Understanding (2026.findings-acl)

Challenge:	Recent advances in code generation focus on optimizing the thought process, but lack effective process supervision, making it difficult to optimize the thoughts.
Approach:	They propose a method that leverages the code execution feedback to build a code PRM by collecting a large dataset of thought traces and then training it to take both the reasoning process and code execution as input.
Outcome:	The proposed approach outperforms baselines and strong LLMs in the inference stage.

Copied to clipboard

Zeqi Chen, Zhaoyang Chu, Yi Gui, Feng Guo, Yao Wan, Chuan Shi

CodeEvo: Interaction-Driven Synthesis of Code-centric Data through Hybrid and Iterative Feedback (2026.acl-long)

Challenge:	Existing structure-aware approaches treat structure as serialized text prompts or auxiliary training objectives, failing to provide explicit guidance during inference.
Approach:	They propose a plug-and-play method that enhances Large Language Models with Code Graph information through an external, trainable Bridge module.
Outcome:	The proposed method decouples structural reasoning from textual generation without updating the backbone.

Copied to clipboard

Qiushi Sun, Jingyang Gong, Lei Li, Qipeng Guo, Fei Yuan

Challenge:	Existing methods for generating instruction-code pairs rely on rigid heuristics and are labor-intensive.
Approach:	They propose a dual-agent architecture that integrates a Coder and a Reviewer to orchestrate the generation trajectory.
Outcome:	The proposed architecture outperforms baselines on a large-scale dataset of instruction-code pairs with stepped difficulty levels.

CDS: Data Synthesis Method Guided by Cognitive Diagnosis Theory (2025.findings-acl)

Copied to clipboard

Haokun Zhao, Jinyi Han, Jiaqing Liang, Yanghua Xiao, Xiaojun Meng, Jiansheng Wei

Linguistics & Cognitive NLP

Challenge:	Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse tasks, but the complexity of emerging tasks and higher performance demands highlight the need for continuous improvement.
Approach:	They propose a method that refines evaluation results and characterizes model profiles at the knowledge component level.
Outcome:	The proposed method improves performance across multiple benchmarks and academic exams.

Does ChatGPT Know That It Does Not Know? Evaluating the Black-Box Calibration of ChatGPT (2024.lrec-main)

Copied to clipboard

Youliang Yuan, Wenxuan Wang, Qingshuo Guo, Yiming Xiong, Chihao Shen, Pinjia He

Natural Language Interfaces

Challenge:	Recent performance of ChatGPT in downstream tasks is questionable, but does it know that it does not know?
Approach:	They propose to use three types of proxy confidence to evaluate ChatGPT's black-box calibration ability.
Outcome:	The proposed model exhibits a positive correlation with accuracy in TruthfulQA and a negative correlation in the ModAr dataset.

PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Finance (2022.emnlp-main)

Copied to clipboard

Yang Deng, Wenqiang Lei, Wenxuan Zhang, Wai Lam, Tat-Seng Chua

Natural Language Interfaces

Challenge:	Existing studies on financial question answering systems focus on passively responding to user queries.
Approach:	They propose a new dataset to facilitate conversational question answering over hybrid contexts in finance . they propose PACIFIC to combine clarification question generation and CQA .
Outcome:	The proposed method performs multi-task learning over all sub-tasks in PACIFIC . it incorporates a simple ensemble strategy to alleviate error propagation issue .

CodeArena: Evaluating and Aligning CodeLLMs on Human Preference (2025.emnlp-main)

Copied to clipboard

Jian Yang, Jiaxi Yang, Wei Zhang, Jin Ke, Yibo Miao, Lei Zhang, Liqun Yang, Zeyu Cui, Yichang Zhang, Zhoujun Li, Binyuan Hui, Junyang Lin

Visual Modeling of Turkish Morphology (2020.lrec-1)

Challenge:	Code large language models (codeLLMs) focus on synthesizing the correct code snippet, ignoring the alignment with human preferences.
Approach:	They propose a benchmark code-based on 40 categories and 44 programming languages to emulate real-world coding tasks.
Outcome:	The proposed benchmarks show that open-source code LLMs perform better than open-sourced ones.

Copied to clipboard

Berke Özenç, Ercan Solak

ReflectionCoder: Learning from Reflection Sequence for Enhanced One-off Code Generation (2025.acl-long)

Challenge:	Currently, there are three publicly accessible morphological analyzers for Turkish .
Approach:	They propose to make modeling easier and more maintainable by using diagramming tools and automating much of the code generation.
Outcome:	The proposed model can be easily maintained and the code generation automated.

Copied to clipboard

Houxing Ren, Mingjie Zhan, Zhongyuan Wu, Aojun Zhou, Junting Pan, Hongsheng Li

Focused-DPO: Enhancing Code Generation Through Focused Preference Optimization on Error-Prone Points (2025.findings-acl)

Challenge:	Existing methods to enhance code generation performance include integrating compiler feedback.
Approach:	They propose a method that integrates compiler feedback to improve one-off code generation performance.
Outcome:	The proposed method improves one-off code generation performance on three benchmarks and can be applied to other domains that focus on final results and require long reasoning paths.

Copied to clipboard

Kechi Zhang, Ge Li, Jia Li, Yihong Dong, Jia Li, Zhi Jin

Learning from Evolving Training Dynamics: An Entropy-Maximizing Data Curation Strategy for LLM Supervised Post-Training (2026.acl-long)

Challenge:	Current code generation models produce errors concentrated at specific error-prone points, affecting accuracy of code.
Approach:	They propose a framework that focuses preference optimization on error-prone areas . focused-DPO improves the accuracy and reliability of code generation by reducing common errors .
Outcome:	The proposed framework improves code generation by focusing on error-prone areas.

Copied to clipboard

Mengxiang Zhang, Lingyuan Liu

Challenge:	EVO-Curate is a dynamic data curation framework that synchronizes sample complexity with the maturing capacity of the Large Language Model (LLM).
Approach:	They propose a dynamic data curation framework that synchronizes sample complexity with maturing capacity of the Large Language Model (LLM) they use an Adaptive Dynamics Measurer to synthesize instantaneous difficulty and historical variability into a multidimensional utility score.
Outcome:	The proposed framework outperforms standard training baselines and traditional CL methods on instruction following, mathematical reasoning, and code generation architectures while maintaining manageable computational overhead.

A Reward-Guided Dual-Phase Framework for Adaptive Inference-Time Reasoning (2026.findings-acl)

Copied to clipboard

Yingqian Cui, Zhenwei Dai, Pengfei He, Bing He, Hui Liu, Zhan Shi, Xianfeng Tang, Jingying Zeng, Suhang Wang, Yue Xing, Jiliang Tang, Benoit Dumoulin

Enhancing Code Generation Performance of Smaller Models by Distilling the Reasoning Ability of LLMs (2024.lrec-main)

Challenge:	Large Language Models (LLMs) have made strong progress in reasoning.
Approach:	They propose a dual-phase test-time scaling framework that separates planning and execution and performs search over each phase independently.
Outcome:	Experiments on math reasoning and code generation benchmarks show that the proposed approach improves accuracy while reducing redundant computation.

Copied to clipboard

Zhihong Sun, Chen Lyu, Bolun Li, Yao Wan, Hongyu Zhang, Ge Li, Zhi Jin

Generating Data for Symbolic Language with Large Language Models (2023.emnlp-main)

Challenge:	Large Language Models (LLMs) have made significant advances in code generation through the ‘Chain-of-Thought’ prompting technique.
Approach:	They propose a framework which aims to transfer LLMs’ reasoning capabilities to smaller models through distillation.
Outcome:	The proposed framework improves the smaller model's code generation performance by over 130% on the APPS benchmark.

Copied to clipboard

Jiacheng Ye, Chengzu Li, Lingpeng Kong, Tao Yu

PERC: Plan-As-Query Example Retrieval for Underrepresented Code Generation (2025.coling-main)

Challenge:	Large language models (LLMs) bring performance and complexity, but they incur a large computational cost in practice.
Approach:	They propose a task-based model which uses large language models to generate symbolic language data by an informative prompt and agreement-based verifier.
Outcome:	The proposed model can generate symbolic language data with a few human demonstrations and saves a considerable amount of inference effort.

Copied to clipboard

Jaeseok Yoo, Hojae Han, Youngwon Lee, Jaejin Kim, Seung-won Hwang

Semantic Captioning: Benchmark Dataset and Graph-Aware Few-Shot In-Context Learning for SQL2Text (2025.coling-main)

Challenge:	Using large language models to generate code has shown significant promise, but selecting effective examples to improve generation quality remains a challenging task.
Approach:	They propose a framework that utilizes algorithmic plans to identify and retrieve effective examples.
Outcome:	The proposed framework outperforms the state-of-the-art RAG methods in code generation even when the source and target languages match or differ.

Copied to clipboard

Ali Al Lawati, Jason Lucas, Prasenjit Mitra

CodeIP: A Grammar-Guided Multi-Bit Watermark for Large Language Models of Code (2024.findings-emnlp)

Challenge:	Large Language Models (LLMs) have shown remarkable performance in various NLP tasks, including semantic parsing, which translates natural language into formal code representations.
Approach:	They propose a semantic captioning task to repurpose semantic parsing datasets for semantic captions.
Outcome:	The proposed model outperforms random selection and other methods by 39% on BLEU score.

Copied to clipboard

Batu Guan, Yao Wan, Zhangqian Bi, Zheng Wang, Hongyu Zhang, Pan Zhou, Lichao Sun

EpiGEN: An Efficient Multi-Api Code GENeration Framework under Enterprise Scenario (2024.lrec-main)

Challenge:	Large Language Models (LLMs) have shown significant potential in code generation, but they also present challenges regarding the protection of Intellectual Property (IP) related to model architectures, weights, and training data.
Approach:	They propose a multi-bit watermarking technique that embeds additional information to preserve provenance details, such as the vendor ID of an LLM.
Outcome:	The proposed technique preserves provenance details while maintaining syntactical correctness of generated code.

Copied to clipboard

Sijie Li, Sha Li, Hao Zhang, Shuyang Li, Kai Chen, Jianyong Yuan, Yi Cao, Lvqing Yang

Self-play through Computational Runtimes improves Chart Reasoning (2025.findings-acl)

Challenge:	Existing approaches to large language models fail to meet expectations for code generation tasks . existing approaches are faced with drawbacks of high resource consumption and inadequate handling of multi-API tasks.
Approach:	They propose an Efficient multi-Api code GENeration framework that uses private APIs to pre-train LLMs.
Outcome:	The proposed framework shows good acceptability and readability on single-GPU tasks compared to fully fine-tuned LLMs with a larger number of parameters.

Copied to clipboard

Tautvydas Misiūnas, Hassan Mansoor, Jasper Uijlings, Oriana Riva, Victor Carbune

Large Language Models for IT Automation Tasks: Are We There Yet? (2026.findings-acl)

Challenge:	Vision-language models (VLMs) achieve impressive zero-shot performance on multimodal reasoning tasks.
Approach:	They propose a self-play programming interface which leverages visual reasoning models to generate code to decompose a complex visual reasoning task in sub-tasks and use it as a tool to solve decomposed tasks.
Outcome:	The proposed approach outperforms baselines on difficult chart reasoning benchmarks.

Copied to clipboard

Md. Mahadi Hassan, John Salvador, Akond Ashfaque Ur Rahman, Santu Karmaker

BeamLoRA: Beam-Constraint Low-Rank Adaptation (2025.acl-long)

Challenge:	Existing benchmarks rely on synthetic tasks that fail to capture the needs of practitioners who use IT automation tools.
Approach:	They evaluate 14 open-source and 3 proprietary LLMs and find that GPT-4.1-Mini achieves the best pass@10 rate of 23.9%, while Claude-3.5-Sonnet achieves best pass @1 performance.
Outcome:	The evaluated LLMs perform poorly in 126 tasks and show that they lack state reconciliation capabilities and lack module knowledge.

Copied to clipboard

Naibin Gu, Zhenyu Zhang, Xiyu Liu, Peng Fu, Zheng Lin, Shuohuan Wang, Yu Sun, Hua Wu, Weiping Wang, Haifeng Wang

DebateCoder: Towards Collective Intelligence of LLMs via Test Case Driven LLM Debate for Code Generation (2025.acl-long)

Challenge:	Low-Rank Adaptation (LoRA) is one of the most efficient parameter-efficient fine-tuning methods.
Approach:	They propose to conceptualize each LoRA module as a beam where each rank corresponds to a potential sub-solution.
Outcome:	The proposed method improves performance on three base models and 12 datasets.

Copied to clipboard

Jizheng Chen, Kounianhua Du, Xinyi Dai, Weiming Zhang, Xihuai Wang, Yasheng Wang, Ruiming Tang, Weinan Zhang, Yong Yu

Exploring Hybrid Question Answering via Program-based Prompting (2024.acl-long)

Challenge:	Existing debate-based approaches to code generation are limited due to several reasons: 1) Reliance on different instances of the same LLM for debate, 2) under-utilization of test cases, and 3) reliance on third-party moderators for result consolidation and decision-making.
Approach:	They propose to use test cases to analyze code and identify bugs while opposing models generate test cases for each other to challenge each other's code during the debate process.
Outcome:	The proposed model collects intelligence of LLMs via test case-driven debate for code generation.

Copied to clipboard

Qi Shi, Han Cui, Haofeng Wang, Qingfu Zhu, Wanxiang Che, Ting Liu

GiFT: Gibbs Fine-Tuning for Code Generation (2025.acl-long)

Challenge:	Existing approaches to question answering over heterogeneous data are limited due to large scale of information and organic coupling of heterogenous data.
Approach:	They propose a program-based prompting framework for hybrid question answering tasks . it integrates various functions to perform hybrid information-seeking over data .
Outcome:	The proposed framework surpasses baseline systems and achieves the best performance under the fewshot settings.

Copied to clipboard

Haochen Li, Wanjin Feng, Xin Zhou, Zhiqi Shen

Multitask-Bench: Unveiling and Mitigating Safety Gaps in LLMs Fine-tuning (2025.coling-main)

Challenge:	Training Large Language Models (LLMs) with synthetic data is a prevalent practice in code generation.
Approach:	They propose a method to fine-tune large language models with code drawn from a conditional distribution, conditioned on a specific seed description.
Outcome:	The proposed method improves performance on four datasets and shows that it can be used to fine-tune LLMs with code derived from the marginal distribution.

Copied to clipboard

Essa Jan, Nouar Aldahoul, Moiz Ali, Faizan Ahmad, Fareed Zaffar, Yasir Zaki

JumpCoder: Go Beyond Autoregressive Coder via Online Modification (2024.acl-long)

Challenge:	Recent advances in Large Language Models (LLMs) have led to their adoption across a wide range of tasks, ranging from code generation to machine translation and sentiment analysis.
Approach:	They propose to fine-tune LLMs on benign (non-harmful) data to ensure safe outputs.
Outcome:	The proposed model reduces attack success rates across a range of tasks without compromising its usefulness.

Copied to clipboard

Mouxiang Chen, Hao Tian, Zhongxin Liu, Xiaoxue Ren, Jianling Sun

Challenge:	Existing code large language models lack reversibility and autoregressive sequential generation is incapable of correcting previous missing statements as humans do.
Approach:	They propose a model-agnostic framework that enables human-like online modification and non-sequential generation to augment code large language models.
Outcome:	The proposed framework enables human-like modification and non-sequential generation to augment code large language models.

Planning-Driven Programming: A Large Language Model Programming Workflow (2025.acl-long)

Copied to clipboard

Chao Lei, Yanchuan Chang, Nir Lipovetzky, Krista A. Ehinger

Layer-Level Self-Exposure and Patch: Affirmative Token Mitigation for Jailbreak Attack Defense (2025.naacl-long)

Challenge:	Recent research suggests continuous program refinements through visible tests to improve code generation accuracy in large language models (LLMs).
Approach:	They propose an LLM programming workflow to improve both initial code generation and subsequent refinements within a structured two-phase workflow.
Outcome:	The proposed workflow improves both initial code generation and subsequent refinements within a structured two-phase workflow.

Copied to clipboard

Yang Ouyang, Hengrui Gu, Shuhang Lin, Wenyue Hua, Jie Peng, Bhavya Kailkhura, Meijun Gao, Tianlong Chen, Kaixiong Zhou

ExecVerify: White-Box RL with Verifiable Stepwise Rewards for Code Execution Reasoning (2026.acl-long)

Challenge:	Existing methods to defend against jailbreak attacks exploit vulnerabilities to elicit unintended or harmful outputs.
Approach:	They propose a method to defend against jailbreak attacks by patching specific layers within large language models through self-augmented datasets.
Outcome:	The proposed approach reduces harmfulness and attack success rate of jailbreak attacks without compromising utility for benign queries compared to previous methods.

Copied to clipboard

Lingxiao Tang, He Ye, Zhaoyang Chu, Muyang Ye, Zhongxin Liu, Xiaoxue Ren, Lingfeng Bao

Explainability & Interpretability in NLP

Challenge:	Existing methods for code execution reasoning are limited by the difficulty of the training data.
Approach:	They propose a model that uses reinforcement learning to reward correct answers from execution traces.
Outcome:	The proposed model improves pass@1 by up to 5.9% on code generation tasks over strong baselines.

Integrate the Essence and Eliminate the Dross: Fine-Grained Self-Consistency for Free-Form Language Generation (2024.acl-long)

Copied to clipboard

Xinglin Wang, Yiwei Li, Shaoxiong Feng, Peiwen Yuan, Boyuan Pan, Heda Wang, Yao Hu, Kan Li

Is GPT-4 a Good Data Analyst? (2023.findings-emnlp)

Challenge:	Existing methods to improve output quality without aggregating input tokens are limited by the complexity of aggregation of responses.
Approach:	They propose to extract and integrate segment-level commonalities from candidate samples to enhance performance of LLMs in open-ended and reasoning tasks.
Outcome:	The proposed method improves performance on reasoning, code generation and mathematical reasoning tasks without requiring additional models and overlooking the knowledge present among the candidates.

Copied to clipboard

Liying Cheng, Xingxuan Li, Lidong Bing

ModeX: Evaluator-Free Best-of-N Selection for Open-Ended Generation (2026.acl-long)

Challenge:	Large language models (LLMs) have shown their powerful capabilities in plenty of domains and tasks, including context understanding, code generation, language generation, data storytelling, etc.
Approach:	They propose to use GPT-4 as a data analyst to perform end-to-end data analysis with databases from a wide range of domains.
Outcome:	The proposed framework compares GPT-4 with human data analysts to perform end-to-end data analysis with databases from a wide range of domains.

Copied to clipboard

Hyeong Kyu Choi, Sharon Li

Speculative Decoding for Multi-Sample Inference (2025.findings-emnlp)

Challenge:	Existing approaches to selecting a single high-quality output from multiple generations are limited in their applicability and efficiency.
Approach:	They propose a method that generalizes majority voting to open-ended text generation . modeX leverages structural information across multiple generation paths to select a "modal" output .
Outcome:	The proposed framework outperforms standard single- and multi-path baselines in open-ended tasks.

Copied to clipboard

Yiwei Li, Jiayi Shi, Shaoxiong Feng, Peiwen Yuan, Xinglin Wang, Yueqi Zhang, Ji Zhang, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li

Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning (2023.emnlp-main)

Challenge:	Speculative decoding method exploits consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or external databases.
Approach:	They propose a speculative decoding method that exploits the consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or external databases.
Outcome:	The proposed method exploits the intrinsic consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or databases.

Copied to clipboard

Saibo Geng, Martin Josifoski, Maxime Peyrard, Robert West

Revisiting the Impact of Pursuing Modularity for Code Generation (2024.findings-emnlp)

Challenge:	Existing grammar-constrained decoding methods are limited to specific tasks . a grammar constraint is used to control the generation of LMs, but it is limited to a few tasks a task is not performed.
Approach:	They propose grammar-constrained decoding to control the generation of large language models . they demonstrate that grammars can describe the output space for a wider range of tasks .
Outcome:	The proposed grammars outperform unconstrained models on information extraction, entity disambiguation, and constituency parsing.

Copied to clipboard

Deokyeong Kang, KiJung Seo, Taeuk Kim

Beyond Superficial Tests: Adversarial Refinement for Reliable Property-Based Testing (2026.findings-acl)

Challenge:	a recent study examines the impact of modularity on code generation in large language models . modularity is not a core factor for improving performance of code generation models, argues a new study .
Approach:	They introduce a new metric to measure the impact of modularity in code generation . they find modularity is not a core factor for improving performance of LLMs .
Outcome:	The proposed metric shows that modularity is not a core factor for improving performance . coding assistants are becoming increasingly essential for programmers .

Copied to clipboard

Xiao Li, Runlin Liu, Zhe Zhang, Xiang Gao, Hailong Sun

Compute Optimal Scaling of Skills: Knowledge vs Reasoning (2025.findings-acl)

Challenge:	Large Language Models (LLMs) have demonstrated remarkable proficiency in code generation, yet their application to Property-Based Testing (PBT) remains fraught with a superficiality gap.
Approach:	They propose an agentic framework that hardens software properties through Adversarial Refinement.
Outcome:	a new framework hardens software properties through Adversarial Refinement that detects and fixes bugs in top-tier libraries.

Copied to clipboard

Nicholas Roberts, Niladri S. Chatterji, Sharan Narang, Mike Lewis, Dieuwke Hupkes

Dynamics of Instruction Fine-Tuning for Chinese Large Language Models (2025.coling-main)

Challenge:	Scaling laws are a critical component of the LLM development pipeline, but little is known about whether the COs of individual skills such as mathematical reasoning, question answering (QA) or coding, align with these APEs.
Approach:	They examine knowledge-based QA and code generation to find out whether skill-dependent scaling is an artefact of the pretraining datamix.
Outcome:	The proposed scaling laws are skill-dependent, and knowledge and code exhibit fundamental differences in scaling behaviour when corrected for datamix differences.

Copied to clipboard

Chiyu Song, Zhanchao Zhou, Jianhao Yan, Yuejiao Fei, Zhenzhong Lan, Yue Zhang

MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization (2024.findings-acl)

Challenge:	Instruction tuning is a burgeoning method to elicit the general intelligence of Large Language Models.
Approach:	They investigate the effects of data quantity, model size, and data construction methods on instruction tuning for Chinese LLMs.
Outcome:	The proposed model includes over 40,000 high-quality instruction instances covering ten underlying abilities.

Copied to clipboard

Zhiyu Yang, Zihan Zhou, Shuo Wang, Xin Cong, Xu Han, Yukun Yan, Zhenghao Liu, Zhixing Tan, Pengyuan Liu, Dong Yu, Zhiyuan Liu, Xiaodong Shi, Maosong Sun

Extracting the Essence and Discarding the Dross: Enhancing Code Generation with Contrastive Execution Feedback (2025.coling-main)

Challenge:	Scientific data visualization is an essential process in research, but its use of large language models remains unexplored.
Approach:	They propose a model-agnostic LLM agent framework to automate scientific data visualization tasks.
Outcome:	The proposed framework improves performance of commercial and open-source models.

Copied to clipboard

Xuanyu Zhang, Qing Yang

Teaching Your Models to Understand Code via Focal Preference Alignment (2025.emnlp-main)

Challenge:	erroneous code generation methods amalgamate feedback and correct code as target sentences . a new approach to code generation with feedback is needed to improve model performance .
Approach:	They propose a learning-based code generation model with execution feedback that integrates feedback and correct code as target sentences.
Outcome:	a new model with execution feedback shows improvements in generating accurate code and understanding error correction.

Copied to clipboard

Jie Wu, Haoling Li, Xin Zhang, Xiao Liu, Yangyu Huang, Jianwen Luo, Yizhen Zhang, Zuchao Li, Ruihang Chu, Yujiu Yang, Scarlett Li

CodeFusion: A Pre-trained Diffusion Model for Code Generation (2023.emnlp-main)

Challenge:	Existing methods for supervised fine-tuning focus on unit test feedback to construct preference pairs.
Approach:	They propose a preference alignment framework that mimics human iterative debugging to refine Code LLMs.
Outcome:	Experiments show that Preference Learning improves on BigCodeBench and BigCodeBind tasks.

Copied to clipboard

Mukul Singh, José Cambronero, Sumit Gulwani, Vu Le, Carina Negreanu, Gust Verbruggen

Process-Supervised Reinforcement Learning for Code Generation (2025.emnlp-main)

Challenge:	Existing models for code generation from natural language do not allow reconsidering earlier tokens . prior work has explored grouped beam search or nucleus sampling to generate diverse text.
Approach:	They propose a diffusion code generation model that iteratively denoises a program conditioned on the encoded natural language.
Outcome:	The proposed model outperforms state-of-the-art models in accuracy and diversity compared to existing models.

Copied to clipboard

Yufan Ye, Ting Zhang, Wenbin Jiang, Hua Huang

MultiCodeAttack: Iterative Jailbreak Attacking on LLMs with Multi-Code Prompt Injection (2026.findings-acl)

Challenge:	Existing reinforcement learning strategies based on outcome supervision have shown effectiveness in code generation tasks, but their effectiveness in the field of code generation remains limited.
Approach:	They propose a method that uses a teacher model to mutate and refactor statements and a compiler to automatically label them.
Outcome:	The proposed method improves performance in complex code generation tasks.

Copied to clipboard

Weifeng Sun, Meng Yan, Zhou Yang, Yuchen Chen, Song Sun, David Lo

ArchCode: Incorporating Software Requirements in Code Generation with Large Language Models (2024.acl-long)

Challenge:	Existing approaches to jailbreak rely on fixed template design and a single programming language . however, existing approaches do not consider language diversity or adaptive template evolution .
Approach:	They propose a structured jailbreak framework that explores and optimizes multi-language code templates.
Outcome:	The proposed framework outperforms existing jailbreak baselines and produces higher harmful outputs than baseline methods.

Copied to clipboard

Hojae Han, Jaejin Kim, Jaeseok Yoo, Youngwon Lee, Seung-won Hwang

Case2Code: Scalable Synthetic Data for Code Generation (2025.coling-main)

Challenge:	Despite the critical role of software requirements, these criteria have not been studied actively in previous code generation works.
Approach:	They propose a framework that leverages in-context learning to organize and extrapolate unexpressed requirements from textual descriptions.
Outcome:	The proposed framework generates functional requirements from textual descriptions and extrapolates unexpressed requirements from them.

Copied to clipboard

Yunfan Shao, Linyang Li, Yichuan Ma, Peiji Li, Demin Song, Qinyuan Cheng, Shimin Li, Xiaonan Li, Pengyu Wang, Qipeng Guo, Hang Yan, Xipeng Qiu, Xuanjing Huang, Dahua Lin

CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges (2024.acl-long)

Challenge:	Large Language Models (LLMs) have shown outstanding breakthroughs in code generation.
Approach:	They propose a case-to-code induction task that exploits the expressiveness and correctness of programs by incorporating LLMs into their training.
Outcome:	The proposed task improves distribution case-to-code induction and various coding generation tasks.

Copied to clipboard

Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, Zhi Jin

Low-Entropy Watermark Detection via Bayes’ Rule Derived Detector (2025.findings-acl)

Challenge:	Large Language Models excel in simple tasks such as generating standalone code units, but real-world software development often involves complex code repositories with complex dependencies and extensive documentation.
Approach:	They propose a novel LLM-based agent framework that employs external tools for effective repo-level code generation.
Outcome:	The proposed framework outperforms commercial products like Github Copilot in the humanEval benchmark and shows that it is adaptable and efficient across multiple code generation tasks.

Copied to clipboard

Beining Huang, Du Su, Fei Sun, Qi Cao, Huawei Shen, Xueqi Cheng

Numerical Reasoning

Challenge:	Existing methods for text watermarking ignore strong evidences embedded in low-entropy tokens, causing statistical measures to falsely indicate the absence of a watermark.
Approach:	They propose a Bayes' Rule derived watermark Detector which exploits watermark information from every token by leveraging the posterior probability of watermark’s presence.
Outcome:	The proposed method achieves 50% and 70% relative improvements over baselines in code generation and math problem-solving tasks.

Efficient Beam Search for Large Language Models Using Trie-Based Decoding (2025.emnlp-main)

Copied to clipboard

Brian J Chan, Mao-xun Huang, Jui-Hung Cheng, Chao-Ting Chen, Hen-Hsen Huang

Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models (2024.acl-long)

Challenge:	Large language models (LLMs) face memorybound performance bottlenecks due to their high memory requirements.
Approach:	They propose a trie-based parallel decoding method that shares a single KV cache across beams with common prefixes to dramatically reduce memory usage and enables efficient decoding.
Outcome:	The proposed method significantly reduces memory usage and enables efficient decoding without compromising generation quality.

Copied to clipboard

Martin Riddell, Ansong Ni, Arman Cohan

OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement (2024.findings-acl)

Challenge:	Recent studies have shown that large language models are contaminated with data from pretraining and finetuning tasks.
Approach:	They perform extensive analysis on the factors that affect model memorization and generalization, such as model size, problem difficulty, and question length.
Outcome:	The results show that models perform better on the subset of the benchmarks where similar solutions are seen during training.

Copied to clipboard

Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, Xiang Yue

CodeDPO: Aligning Code Models with Self Generated and Verified Source Code (2025.acl-long)

Challenge:	OpenCodeInterpreter-33B provides a high level of performance for code generation, executing, and iterative refinement.
Approach:	They propose a family of open-source code systems for generating, executing, and iteratively refining code.
Outcome:	The OpenCodeInterpreter-33B performs well on humanEval, MBPP, and EvalPlus benchmarks.

Copied to clipboard

Kechi Zhang, Ge Li, Yihong Dong, Jingjing Xu, Jun Zhang, Jing Su, Yongfei Liu, Zhi Jin

ReCode: Robustness Evaluation of Code Generation Models (2023.acl-long)

Challenge:	Existing training methods for code generation do not improve code correctness and efficiency.
Approach:	They propose a framework that integrates preference learning into code generation to improve code correctness and efficiency.
Outcome:	The proposed framework improves code correctness and efficiency by integrating preference learning into code generation.

Copied to clipboard

Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, Bing Xiang

Python Code Generation by Asking Clarification Questions (2023.acl-long)

Challenge:	Existing work on robustness in text or code tasks has focused on classification, while robustness for code generation tasks is an uncharted area.
Approach:	They propose a robustness evaluation benchmark for code generation models that customizes over 30 transformations specifically for code on docstrings, function and variable names, code syntax, and code format.
Outcome:	The proposed model performs better on human annotators and on SOTA models with human annnotators.

Copied to clipboard

Haau-Sing (Xiaocheng) Li, Mohsen Mesgar, André Martins, Iryna Gurevych

SecCoder: Towards Generalizable and Robust Secure Code Generation (2024.emnlp-main)

Challenge:	Recent work addresses text-to-code generation using pretrained language models (PLMs) for large-scale NLD: Logistic Regression.
Approach:	They propose a dataset containing pairs of natural language descriptions and code with created synthetic clarification questions and answers to solve the under-specified nature of a natural language description.
Outcome:	The proposed model improves on previous models, while introducing new challenges to the community, including when and what clarification questions should be asked.

Copied to clipboard

Boyu Zhang, Tianyu Du, Junkai Tong, Xuhong Zhang, Kingsum Chow, Sheng Cheng, Xun Wang, Jianwei Yin

Better Language Models of Code through Self-Improvement (2023.findings-acl)

Challenge:	Existing secure code generation methods have limited generalizability to unseen test cases and poor robustness against the attacked model, leading to safety failures in code generation.
Approach:	They propose a generalizable and robust secure code generation method SecCoder by using in-context learning and the safe demonstration.
Outcome:	The proposed method achieves a significant security improvement of 7.20% on unseen test cases and better robustness against the attacked model.

Copied to clipboard

Hung To, Nghi Bui, Jin L.C. Guo, Tien Nguyen

ByteSized32: A Corpus and Challenge Task for Generating Task-Specific World Models Expressed as Text Games (2023.emnlp-main)

Challenge:	Pre-trained language models for code (PLMCs) are pre-taught on large datasets with multi-modal objectives, but fine-tuning them requires extensive supervision and is limited by the size of the dataset provided.
Approach:	They propose a data augmentation framework that utilizes knowledge from the pre-training and fine-tuning stage to augment training data, which is then used for the next step.
Outcome:	The proposed framework significantly improves pre-trained language models’ performance in sequence-generation tasks, such as code summarization and code generation in the CodeXGLUE benchmark.

Copied to clipboard

Ruoyao Wang, Graham Todd, Xingdi Yuan, Ziang Xiao, Marc-Alexandre Côté, Peter Jansen

CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code (2023.emnlp-main)

Challenge:	We show that language models can generate explicit, interpretable, and interactive world models of scientific and common-sense reasoning tasks.
Approach:	They propose a corpus of 32 reasoning-focused text games expressed as hundreds of lines of Python code to facilitate this task.
Outcome:	The proposed games can generate runnable games on unseen topics in 28% of cases.

Copied to clipboard

Shuyan Zhou, Uri Alon, Sumit Agarwal, Graham Neubig

Re-ReST: Reflection-Reinforced Self-Training for Language Agents (2024.emnlp-main)

Challenge:	NLCode generates long expressions and statements rather than a single next-token . evaluating and comparing different models has remained a challenge .
Approach:	They propose a code-generating evaluation metric built on BERTScore . they use five language-specific pretrained models to evaluate their code .
Outcome:	The proposed evaluation metric achieves higher correlation with human preference and functional correctness than existing metrics across four programming languages.

Copied to clipboard

Zi-Yi Dou, Cheng-Fu Yang, Xueqing Wu, Kai-Wei Chang, Nanyun Peng

Natural Language Interfaces

Challenge:	Existing methods to fine tune language agents with reasoning-action trajectories require high-quality model-generated samples, which are hard to obtain for challenging language agent tasks.
Approach:	They propose a method to employ reflection during inference without ground-truth feedback to improve agents more autonomously.
Outcome:	The proposed method improves self-training performance on open-source language agents by 7.6% and 14.1% respectively.

DOS: Dependency-Oriented Sampler for Masked Diffusion Language Models (2026.findings-acl)

Copied to clipboard

Xueyu Zhou, Yangrong Hu, Jian Huang

From Words to Wires: Generating Functioning Electronic Devices from Natural Language Descriptions (2023.findings-emnlp)

Challenge:	Existing decoding strategies for pre-trained MDLMs rely on token-level uncertainty criteria, while largely overlooking sequence-level information and inter-token dependencies.
Approach:	They propose a training-free decoding strategy that leverages inter-token dependencies to inform token updates during generation.
Outcome:	Empirical results show that the proposed approach consistently achieves superior performance on both code generation and mathematical reasoning tasks.

Copied to clipboard

Peter Jansen

S*: Test Time Scaling for Code Generation (2025.findings-emnlp)

Challenge:	Recent work shows that language models have a capacity for electronic circuit design from high-level textual descriptions, akin to code generation.
Approach:	They propose two benchmarks to assess a model's ability to design common microcontroller circuits and code in the Arduino ecosystem.
Outcome:	The proposed model knowledge of electrical components and the model's ability to generate full circuits and code in the Arduino ecosystem are evaluated.

Copied to clipboard

Dacheng Li, Shiyi Cao, Chengkun Cao, Xiuyu Li, Shangyin Tan, Kurt Keutzer, Jiarong Xing, Joseph E. Gonzalez, Ion Stoica

StoryCoder: Narrative Reformulation for Structured Reasoning in LLM Code Generation (2026.acl-long)

Challenge:	S* is the first hybrid test-time scaling framework that significantly improves the coverage and selection accuracy of generated code.
Approach:	They propose a hybrid test-time scaling framework that augments parallel scaling with sequential scaling to further increase the performance.
Outcome:	The proposed framework outperforms existing scaling approaches in large-scale modeling and reasoning models.

Copied to clipboard

Geonhui Jang, Dongyoon Han, YoungJoon Yoo

Why Stop at One Error? Benchmarking LLMs as Data Science Code Debuggers for Multi-Hop and Multi-Bug Errors (2025.emnlp-main)

Challenge:	Existing approaches augment reasoning steps or inject specific structure into how models think, but leave scattered problem conditions unchanged.
Approach:	They propose a narrative reformulation framework that transforms code generation questions into coherent natural language narratives.
Outcome:	The proposed framework improves the performance of 11 code generation models on HumanEval, LiveCodeBench, and CodeForces.

Copied to clipboard

Zhiyu Yang, Shuo Wang, Yukun Yan, Yang Deng

Revisit Self-Debugging with Self-Generated Tests for Code Generation (2025.acl-long)

Challenge:	Recent advances in Large Language Models (LLMs) have significantly reshaped software development practices, particularly in automating code generation and debugging.
Approach:	They propose to evaluate LLMs' capabilities on multi-hop error tracing and multi-bug detection in data science code debugging.
Outcome:	DSDBench adapts datasets from existing data science task benchmarks, such as DABench and MatPlotBench, featuring realistic data science debugging tasks with automatically synthesized multi-hop, multi-bug code snippets.

Copied to clipboard

Xiancai Chen, Zhengwei Tao, Kechi Zhang, Changzhi Zhou, Xinyu Zhang, Wanli Gu, Yuanpeng He, Mengdi Zhang, Xunliang Cai, Haiyan Zhao, Zhi Jin

Plug-and-Play Data Module for Code RL: Adaptive Ambiguity Replay (2026.findings-acl)

Challenge:	Large language models (LLMs) have made significant advances in code generation, but they still face challenges when tackling complex programming tasks beyond their basic capabilities.
Approach:	They propose to integrate self-generated tests into the code generation process . they propose to use post-execution and in-exection self-debugging to mitigate test bias .
Outcome:	The proposed method improves the performance of large language models in code generation tasks by leveraging execution feedback from tests.

Copied to clipboard

Jianqing Zhang, Wei Xia, Zhezheng Hao, Hong Wang, Hande Dong, Qiang Lin, Yang Liu, Jian Cao, Qiang Yang

CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks (2026.acl-long)

Challenge:	Existing approaches to reinforcement learning (RL) rely on static, in-epoch metrics that overlook training dynamics, often introducing low-utility or outdated data.
Approach:	They propose a plug-and-play module that prioritizes cross-epoch ambiguous samples to neutralize the noise from stale experiences.
Outcome:	Extensive experiments on nine LLMs show that Adaptive Ambiguity Replay outperforms state-of-the-art baselines on real-world code editing tasks.

Copied to clipboard

Hongchao Jiang, Yiming Chen, Yushi Cao, Hung-yi Lee, Robby T. Tan

VisualEDU: A Benchmark for Assessing Coding and Visual Comprehension through Educational Problem-Solving Video Generation (2025.findings-emnlp)

Challenge:	Large Language Models (LLMs) are increasingly used to judge code, but their reliability remains poorly understood.
Approach:	They propose a benchmark to evaluate Large Language Models as code judges . they find that small reasoning models outperform larger non-reasoning models .
Outcome:	The proposed benchmark evaluates LLM-as-a-Judge models across three coding tasks.

Copied to clipboard

Hao Chen, Tianyu Shi, Pengran Huang, Zeyuan Li, Jiahui Pan, Qianglong Chen, Lewei He

When Instructions Multiply: Measuring and Estimating LLM Capabilities of Multiple Instructions Following (2025.findings-emnlp)

Challenge:	VisualEDU is a benchmark to evaluate VLMs' ability to produce coherent video from text . advanced proprietary models show promise, but struggle with increasing task complexity .
Approach:	VisualEDU is a benchmark to evaluate VLMs' ability to produce coherent video from text . it integrates meta-prompt learning, visual and code feedback, and a drawing toolkit to enhance output quality.
Outcome:	VisualEDU is a benchmark to evaluate VLMs' ability to produce coherent video from text . it integrates meta-prompt learning, visual and code feedback, and a drawing toolkit to improve output quality.

Copied to clipboard

Keno Harada, Yudai Yamazaki, Masachika Taniguchi, Edison Marrese-Taylor, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo

One-to-many testing for code generation from (just) natural language (2024.findings-emnlp)

Challenge:	a large number of languages are increasingly used to evaluate their ability to follow multiple instructions simultaneously.
Approach:	They propose two benchmarks to evaluate LLMs' ability to follow multiple instructions simultaneously . they use many instruction-following eval and style-aware Mostly Basic programming problems .
Outcome:	The proposed models predict performance on unseen instruction combinations and not used during training with 10% error.

Copied to clipboard

Mansi Uniyal, Mukul Singh, Gust Verbruggen, Sumit Gulwani, Vu Le

Free your mouse! Command Large Language Models to Generate Code to Format Word Documents (2024.emnlp-main)

Challenge:	MBPP relies on test cases to generate the right signature, data contamination is a problem . adapted code generation benchmarks allow for the description to be underspecified with respect to syntactic properties of code.
Approach:	They propose a code generation benchmark that allows for the description to be underspecified with respect to syntactic properties of code.
Outcome:	The proposed model removes ambiguity about the semantics of the task from the descriptions and evaluates generated code on multiple sets of assertions to account for ambiguities in the syntax.

Copied to clipboard

Shihao Rao, Liang Li, Jiapeng Liu, Guan Weixin, Xiyan Gao, Bing Lim, Can Ma

M-Ped: Multi-Prompt Ensemble Decoding for Large Language Models (2025.findings-emnlp)

Challenge:	Recent LLMs have significantly improved code generation, making it increasingly accessible to users.
Approach:	They propose an automatic document formatting method, Text-to-Format, driven by various prompting strategies and a high-quality dataset DocFormEval data.
Outcome:	The proposed method improves the efficiency and experience of users in formatting the document and improves document formatting task.

Copied to clipboard

Jiaxin Guo, Daimeng Wei, Yuanchang Luo, Hengchao Shang, Zongyao Li, Jinlong Yang, Zhanglin Wu, Zhiqiang Rao, Shimin Tao, Hao Yang

DeepGuard: Secure Code Generation via Multi-Layer Semantic Aggregation (2026.acl-long)

Challenge:	a new ensemble decoding approach enhances the performance of Large Language Models.
Approach:	They propose a multi-prompt ensemble decoding approach to enhance LLM performance . they submit n variations of prompts with X to LLMs in batch mode to decode and derive probability distributions .
Outcome:	The proposed method improves pass@k rates, LENS metrics and BLEU scores on diverse NLP tasks.

Copied to clipboard

Li Huang, Zhongxin Liu, Yifan Wu, Tao Yin, Dong li, Jichao Bi, Nankun Mu, Hongyu Zhang, Meng Yan

BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models (2025.findings-emnlp)

Challenge:	Large Language Models (LLMs) can replicate insecure patterns from training data.
Approach:	They propose a framework that leverages distributed security-relevant cues by aggregating representations from multiple upper layers via an attention-based module.
Outcome:	Experiments show that the framework improves the secure-and-correct generation rate by 11.9% over baselines.

Copied to clipboard

Xu Huang, Wenhao Zhu, Hanxu Hu, Conghui He, Lei Li, Shujian Huang, Fei Yuan

Code-SPA: Style Preference Alignment to Large Language Models for Effective and Robust Code Debugging (2025.findings-acl)

Challenge:	Existing multilingual benchmarks focus primarily on language understanding tasks.
Approach:	They develop a multi-way multilingual benchmark that measures critical capabilities of large language models across languages.
Outcome:	Extensive experiments on BenchMAX reveal uneven utilization of core capabilities across languages, emphasizing the performance gaps that scaling model size alone does not resolve.

Copied to clipboard

Tengfei Wen, Xuanang Chen, Ben He, Le Sun

MONAQ: Multi-Objective Neural Architecture Querying for Time-Series Analysis on Resource-Constrained Devices (2025.findings-emnlp)

Challenge:	Large language models (LLMs) have demonstrated impressive capabilities in coding tasks like code generation and debugging.
Approach:	They propose a method which aligns noisy code with the well-structured style familiar to LLMs, mitigating the impact of stylistic inconsistencies.
Outcome:	The proposed method improves debugging performance on poorly styled code across the HumanEval, MBPP and EvalPlus datasets.

Copied to clipboard

Patara Trirat, Jae-Gil Lee

Aligned Multi-View Scripts for Universal Chart-to-Code Generation (2026.acl-long)

Challenge:	Recent efforts in hardware-aware neural architecture search (NAS) automate architecture discovery for specific platforms; however, none focus on general time-series analysis with edge deployment.
Approach:	They propose a framework that reformulates NAS into *Multi-Objective Neural Architecture Q*uerying tasks.
Outcome:	Experiments on 15 datasets show that the proposed framework outperforms both handcrafted models and NAS baselines while being more efficient.

Copied to clipboard

Zhihan Zhang, Lizi Liao

LLMs can be easily Confused by Instructional Distractions (2025.acl-long)

Challenge:	Existing methods for chart-to-code generation are largely Python-centric, limiting practical use and overlooking a critical source of supervision.
Approach:	They propose a chart-to-code generation tool that converts a graph image into an executable plotting script.
Outcome:	The proposed method outperforms existing systems and is competitive with proprietary systems.

Copied to clipboard

Yerin Hwang, Yongil Kim, Jahyun Koo, Taegwan Kang, Hyunkyung Bae, Kyomin Jung

E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning (2025.emnlp-main)

Challenge:	Large language models (LLMs) show exceptional skill in instruction following tasks, but can become vulnerable when they are required to disregard instructions.
Approach:	They propose a benchmark to assess LLMs' performance under instructional distraction.
Outcome:	The proposed benchmark categorizes real-world instances of instructional distraction and evaluates LLMs across four instruction tasks: proofreading, rewriting, translation, and style transfer—alongside five input tasks: reasoning, code generation, mathematical reasoning, bias detection, and question answering.

Copied to clipboard

Zihan Liao, Jun Wang, Hang Yu, Lingxiao Wei, Jianguo Li, Jun Wang, Wei Zhang

PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs (2024.findings-emnlp)

Challenge:	Considerable efforts have been and are still being put into increasing the context length of Large Language Models (LLMs)
Approach:	They propose an approach that divides long contexts into chunks, compresses each into soft prompts using a pretrained text encoder, and aligns these representations with a decoder-only LLM via an adapter.
Outcome:	The proposed approach outperforms 8 state-of-the-art methods in effectiveness and efficiency for document summarization and question answering, and achieves the best performance on LongBench v2 among models of comparable size.

Copied to clipboard

Ankit Yadav, Himanshu Beniwal, Mayank Singh

G-LoRA: Global-Local Decoupled Low-Rank Adaptation (2026.findings-acl)

Challenge:	HumanEval and MBPP are two popular benchmarks for Python code generation.
Approach:	They propose a large-scale human evaluation of two popular Python benchmarks . they propose 185 hand-crafted prompts in a balanced representation of 38 programming concepts across diverse difficulty levels.
Outcome:	The proposed benchmarks show a critical bias towards a limited set of programming concepts, neglecting most of the other concepts entirely.

Copied to clipboard

Jiahao Xiong, Yihong Huang, Yihe Liu, Xianming Hu, Hongbo Zhao, Kai Zhang

Graph-GRPO: Stabilizing Multi-Agent Topology Learning via Group Relative Policy Optimization (2026.findings-acl)

Challenge:	Low-Rank Adaptation (LoRA) improves the fine-tuning efficiency and performance of large language models.
Approach:	They propose a low-rank adaptive approach that decomposes update matrix into global and local adapters and assigns them to local and global adapters.
Outcome:	The proposed method achieves up to 2.7% accuracy improvement over LoRA and its variants on commonsense reasoning, mathematical reasoning, and code generation.

Copied to clipboard

Yueyang Cang, Xiaoteng Zhang, Erlu Zhao, Zehua Ji, Yuhang Liu, Yuchen He, Zhiyuan Ning, Chen Yijun, Wenge Que, Li Shi

Exploring Distributional Shifts in Large Language Models for Code Analysis (2023.emnlp-main)

Challenge:	Recent approaches to optimize communication topology rely on single-sample policy gradients with absolute rewards.
Approach:	They propose a topology optimization framework that integrates Group Relative Policy Optimization.
Outcome:	The proposed topology optimization framework outperforms state-of-the-art methods on reasoning and code generation benchmarks.

Copied to clipboard

Shushan Arakelyan, Rocktim Das, Yi Mao, Xiang Ren

EpiCaR: Knowing What You Don’t Know Matters for Better Reasoning in LLMs (2026.acl-long)

Challenge:	Since the late 2000s, researchers have been reporting poor generalization of statistical learning models to new software systems, such as GitHub Copilot, Amazon CodeWhisperer, Replit, etc.
Approach:	They systematically study how three large language models with code capabilities generalize to out-of-domain data.
Outcome:	The proposed model outperforms the existing model for code generation on multiple domains at once.

Copied to clipboard

Jewon Yeom, Jaewon Sok, Seonghyeon Park, Jeongjae Park, Taesup Kim

Uncovering the Impact of Chain-of-Thought Reasoning for Direct Preference Optimization: Lessons from Text-to-SQL (2025.acl-long)

Challenge:	Existing approaches to improving reasoning abilities of large language models incur a significant calibration cost.
Approach:	They propose an epistemic learning problem that integrates reasoning and calibration into an iterative supervised training framework.
Outcome:	The proposed method achieves Pareto-superiority over standard baselines in accuracy and calibration.

Copied to clipboard

Hanbing Liu, Haoyang Li, Xiaokang Zhang, Ruotong Chen, Haiyong Xu, Tian Tian, Qi Qi, Jing Zhang

The Invisible Hand: Unveiling Provider Bias in Large Language Models for Code Generation (2025.acl-long)

Challenge:	Direct Preference Optimization (DPO) is effective in complex reasoning tasks like math word problems and code generation, but Text-to-SQL datasets often include only final answers (gold SQL queries) without detailed CoT solutions.
Approach:	They found that Direct Preference Optimization (DPO) is crucial for unlocking DPO's potential by augmenting Text-to-SQL datasets with synthetic CoT solutions.
Outcome:	The proposed method achieves consistent and significant performance improvements on Text-to-SQL datasets.

Copied to clipboard

Xiaoyu Zhang, Juan Zhai, Shiqing Ma, Qingshuang Bao, Weipeng Jiang, Qian Wang, Chao Shen, Yang Liu

DocCGen: Document-based Controlled Code Generation (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have emerged as the new recommendation engines, surpassing traditional methods in both capability and scope, particularly in code generation.
Approach:	They propose to use a dataset to investigate a new type of bias in Large Language Models for code generation, provider bias, to determine whether the model favors specific providers.
Outcome:	The proposed model favors services from Google and Amazon, but without explicit directives, and can modify input code to incorporate their preferred providers without user requests.

Copied to clipboard

Sameer Pimparkhede, Mehant Kammakomati, Srikanth Tamilselvam, Prince Kumar, Ashok Kumar, Pushpak Bhattacharyya

Localizing Malicious Outputs from CodeLLM (2025.findings-emnlp)

Challenge:	Large language models (LLMs) produce state-of-the-art performance on natural language to code generation for resource-rich general-purpose languages like C++, Java, and Python.
Approach:	They propose a framework that breaks the NL-to-Code generation task into two steps . they use library documentation to detect the correct libraries and schema rules extracted from the documentation to constrain the decoding .
Outcome:	The proposed framework improves different sized language models across all six evaluation metrics, reducing syntactic and semantic errors in structured code.

Copied to clipboard

Mayukh Borana, Junyi Liang, Sai Sathiesh Rajan, Sudipta Chattopadhyay

CoCoST: Automatic Complex Code Generation with Online Searching and Correctness Testing (2024.emnlp-main)

Challenge:	Using FreqRank, we localize malicious components in outputs for triggered inputs and their corresponding backdoor triggers.
Approach:	They propose a mutation-based defense to localize malicious components in LLM outputs and their corresponding backdoor triggers.
Outcome:	The proposed defense has an average attack success rate (ASR) of 86.6% and can localize the backdoor triggers in 98% of cases.

Copied to clipboard

Xinyi He, Jiaru Zou, Yun Lin, Mengyu Zhou, Shi Han, Zejian Yuan, Dongmei Zhang

PECC: Problem Extraction and Coding Challenges (2024.lrec-main)

Challenge:	Existing methods to improve code generation from natural language descriptions are difficult due to complex structure, subtle bugs, and lack of supplementary contents.
Approach:	They propose a framework that enhances complex code generation by online searching for more information with planned queries and correctness testing for code refinement.
Outcome:	The proposed framework improves the quality of complex code generation on the DS-1000 and ClassEval datasets.

Copied to clipboard

Patrick Haller, Jonas Golde, Alan Akbik

Information Extraction & Text Mining

Challenge:	Existing benchmarks evaluate tasks in isolation, yet the extent to which LLMs can understand prose-style tasks, identify the underlying problems, and then generate appropriate code solutions remains mostly unexplored.
Approach:	They propose a benchmark derived from Advent Of Code challenges and Project Euler, which requires LLMs to interpret narrative-embedded problems, extract requirements, and generate executable code.
Outcome:	The proposed benchmarks show that LLMs can understand prose-style tasks, identify underlying problems, and generate appropriate code solutions in a variety of tasks.

CodeJudge: Evaluating Code Generation with Large Language Models (2024.emnlp-main)

Copied to clipboard

Weixi Tong, Tianyi Zhang

Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality (2025.emnlp-main)

Challenge:	Large Language Models (LLMs) have shown promising performance in code generation, but how to reliably evaluate code generated by LLMs remains a challenging problem.
Approach:	They propose a framework that leverages Large Language Models to evaluate the semantic correctness of generated code without the need for test cases.
Outcome:	The proposed framework outperforms existing methods on four code generation datasets and five programming languages.

Copied to clipboard

Yuto Harada, Yusuke Yamauchi, Yusuke Oda, Yohei Oseki, Yusuke Miyao, Yu Takagi

From Understanding to Generation: An Efficient Shortcut for Evaluating Language Models (2025.emnlp-main)

Challenge:	Recent advances in large language models (LLMs) have greatly improved natural language understanding and generation.
Approach:	They train a wide range of base models on a variety of datasets including code generation, mathematical reasoning, and general-domain tasks.
Outcome:	The results show that training–task synergies persist across all models while others vary substantially, emphasizing the importance of model-specific strategies.

Copied to clipboard

Viktor Hangya, Fabian Küch, Darina Gold

Humanity’s Last Code Exam: Can Advanced LLMs Conquer Human’s Hardest Code Competition? (2025.findings-emnlp)

Challenge:	Iterative evaluation of large language models during training can be time- and compute-intensive.
Approach:	They reformulate generative tasks into computationally cheaper NLU alternatives and test their performance correlation between them.
Outcome:	The proposed alternatives reduce evaluation time by 35x compared to NLU benchmarks.

Copied to clipboard

Xiangyang Li, Xiaopeng Li, Kuicai Dong, null Zhangquanhu, Rongju Ruan, Xinyi Dai, Yasheng Wang, Ruiming Tang

ChipSeek: Optimizing Verilog Generation via EDA-Integrated Reinforcement Learning (2026.acl-long)

Challenge:	o4-mini(high) and Gemini-2.5 Pro achieve pass@1 rates of only 15.9% and 11.4%, respectively.
Approach:	They propose a harmonized online–offline sandbox that guarantees fully reproducible evaluation.
Outcome:	The proposed test reflects the advanced reasoning and code generation ability of large language models.

Copied to clipboard

Zhirong Chen, Kaiyan Chang, Zhuolin Li, Cangyuan Li, Xinyang He, Chujie Chen, Mengdi Wang, Haobo Xu, Yinhe Han, Huawei Li, Ying Wang

Evaluating LLMs’ Mathematical and Coding Competency through Ontology-guided Interventions (2025.findings-acl)

Challenge:	Existing approaches to optimize Register-Transfer Level (RTL) code fail to simultaneously optimize functional correctness and hardware efficiency metrics such as Power, Performance, and Area (PPA).
Approach:	They propose a hierarchical reward based reinforcement learning framework that integrates direct feedback from EDA simulators and synthesis tools into a reward mechanism.
Outcome:	The proposed framework integrates direct feedback from EDA simulators and synthesis tools into a hierarchical reward based reinforcement learning framework.

Copied to clipboard

Pengfei Hong, Navonil Majumder, Deepanway Ghosal, Somak Aditya, Rada Mihalcea, Soujanya Poria

TritonBench: Benchmarking Large Language Model Capabilities for Generating Triton Operators (2025.findings-acl)

Challenge:	Current large language models have shown impressive performance on logical reasoning benchmarks . however, the true depth of their competencies and robustness in reasoning tasks remains an open question .
Approach:	They propose a general ontology of perturbations and a semi-automatic method to apply perturbations to arithmetic reasoning and code generation datasets to test their LLMs' capabilities.
Outcome:	The proposed model outperforms existing models on arithmetic reasoning and code generation tasks.

Copied to clipboard

Jianling Li, ShangZhan Li, Zhenye Gao, Qi Shi, Yuxuan Li, Zefan Wang, Jiacheng Huang, WangHaojie WangHaojie, Jianrong Wang, Xu Han, Zhiyuan Liu, Maosong Sun

Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion (2024.emnlp-main)

Challenge:	Triton is a high-level Python-like programming language for building efficient GPU kernels.
Approach:	They propose a TritonBench benchmark that provides a comprehensive evaluation of Tritonic operators on widely deployed GPUs.
Outcome:	The proposed benchmarks show that current LLMs struggle to generate efficient Triton operators on widely deployed GPUs aligned with industry applications.

Copied to clipboard

Yannis Flet-Berliac, Nathan Grinsztajn, Florian Strub, Eugene Choi, Bill Wu, Chris Cremer, Arash Ahmadian, Yash Chandak, Mohammad Azar, Olivier Pietquin, Matthieu Geist

RAR: Retrieval-augmented retrieval for code generation in low resource languages (2024.emnlp-main)

Challenge:	Reinforcement Learning (RL) is a method used to fine tune Large Language Models (LLMs) using a reward model trained from preference data to better align with human judgment.
Approach:	They propose a Reinforcement Learning (RL) algorithm that can estimate the optimal policy even from off-policy data.
Outcome:	The proposed algorithm can estimate the optimal policy even from off-policy data.

Copied to clipboard

Avik Dutta, Mukul Singh, Gust Verbruggen, Sumit Gulwani, Vu Le

Learning from Failures: Error Notebook-guided Secure Code Generation (2026.findings-acl)

Challenge:	Either examples or documentation are commonly used for improved code generation.
Approach:	They propose retrieval augmented retrieval as a two-step method for selecting relevant examples and documentation.
Outcome:	The proposed method outperforms example and grammar retrieval on low-resource languages . it also outperformed two-step retrieval when used independently .

Copied to clipboard

Xinyu Zhong, Peng Lan, Zhifang Liao

EvoHyper: Evolving Hypergraph Topologies for Unified Collaboration in Multi-Agent Communication (2026.findings-acl)

Challenge:	Existing security code generation methods rely on abstract security knowledge, resulting in suboptimal security.
Approach:	They propose a framework that integrates a Security Error Notebook and a Function Erro Notebook to provide concrete, actionable guidance to LLMs.
Outcome:	The proposed framework achieves a substantial leap in SP@1 metric, with GPT-4o-mini performance improving from 60.21% to 66.7% on CodeGuard+.

Copied to clipboard

Heng Zhang, Yihao Zhong, Lubin Gan, Zhihe Chen, Jiajun Wu, Yuling Shi, Xiaodong Gu, Hao Zhang, Haochen You, Jin Huang

Challenge:	Existing methods for multi-agent collaboration use a fixed communication graph and manage collaboration structure and shared memory in separate modules.
Approach:	They propose a framework that uses an evolving hypergraph topology for multi-agent collaboration.
Outcome:	The proposed framework achieves 3.2% to 7.8% accuracy gains over state-of-the-art methods and efficient, reducing token consumption by up to 23.5%.

Identifying & Interactively Refining Ambiguous User Goals for Data Visualization Code Generation (2025.emnlp-main)

Copied to clipboard

Mert Inan, Anthony Sicilia, Alex Xie, Saujas Vaduguru, Daniel Fried, Malihe Alikhani

CURE: Critique-Driven Unified Reinforcement Learning for Test-Time Self-Improvement (2026.acl-long)

Challenge:	ambiguities in natural language can lead to outputs that seem correct but fail to reflect the speaker’s intent.
Approach:	They propose to identify and then resolve ambiguities in natural language and propose metrics to quantify them.
Outcome:	The proposed metrics better correlate with human annotations than uncertainty baselines.

Copied to clipboard

Guirong Chen, Shuqi Ye, Wenkai Yang, Shiqi Shen, Guangyao Shen, Yankai Lin

Challenge:	Existing critique-guided methods fail to equip models with the autonomous improvement capabilities required for test-time scaling.
Approach:	They propose a framework that jointly optimizes a single policy for standard solving, critiquing, and guided re-exploration.
Outcome:	The proposed framework maintains competitive single-turn performance and unlocks effective inference-time scaling.

SpreadNaLa: A Naturalistic Code Generation Evaluation Dataset of Spreadsheet Formulas (2024.lrec-main)

Copied to clipboard

Sebastian Schuster, Ayesha Ansar, Om Agarwal, Vera Demberg

Parallel-SFT: Improving Zero-Shot Cross-Programming-Language Transfer for Code RL (2026.findings-acl)

Challenge:	Existing datasets primarily target the use of code generation models to aid expert programmers in writing code.
Approach:	They propose a natural language code generation model that can translate English descriptions to spreadsheet formulas that can be used to do everyday data processing tasks.
Outcome:	The proposed model performs best among the evaluated methods but generates formulas that differ from human-generated ones.

Copied to clipboard

Zhaofeng Wu, Shiqi Wang, Boya Peng, Anuj Kumar Goyal, Melanie Kambadur, Sebastian Ruder, Yoon Kim, Chloe Bi

TLoRA: Task-aware Low Rank Adaptation of Large Language Models (2026.acl-long)

Challenge:	Modern language models demonstrate impressive coding capabilities in common programming languages (PLs) but their performance in lower-resource PLs is often limited by training data availability.
Approach:	They propose a zero-shot cross-programming-language transfer task for code RL . they propose RL training in a source PL fails to improve performance on other target PLs .
Outcome:	The proposed approach improves transferability in Llama-3.1 code generation on parallel-stack model . it also improves performance on other target PLs, compared to single-PL SFT .

Copied to clipboard

Weicheng Lin, Yi Zhang, Jiawei Dang, Liang-Jie Zhang

LLMs Caught in the Crossfire: Malware Requests and Jailbreak Challenges (2025.acl-long)

Challenge:	Existing low-rank Adaptation (LoRA) methods address only one factor, often at the cost of increased training complexity or reduced practical efficiency.
Approach:	They propose a low-rank Adaptation framework that optimizes initialization and resource allocation at the outset of training.
Outcome:	The proposed framework performs excellently across various tasks while reducing the number of trainable parameters.

Copied to clipboard

Haoyang Li, Huan Gao, Zhiyuan Zhao, Zhiyu Lin, Junyu Gao, Xuelong Li

Search-in-Context: Efficient Multi-Hop QA over Long Contexts via Monte Carlo Tree Search with Dynamic KV Retrieval (2025.findings-acl)

Challenge:	Large Language Models (LLMs) have a high vulnerability to jailbreak attacks that leverage crafted prompts to generate malicious outputs.
Approach:	They propose to use large language models to test their security against jailbreak attacks that leverage crafted prompts to generate malicious outputs.
Outcome:	The proposed model is based on 320 manually crafted malicious code generation requirements, covering 11 jailbreak methods and 29 code functionality categories.

Copied to clipboard

Jiabei Chen, Guang Liu, Shizhu He, Kun Luo, Yao Xu, Jun Zhao, Kang Liu

EGSS: Entropy-guided Stepwise Scaling for Reliable Software Engineering (2026.acl-long)

Challenge:	Existing approaches to multihop question answering (MHQA) over long contexts are often neglecting explicit reasoning or incurring expensive computational costs due to full-attention mechanisms over long contextuals.
Approach:	They propose a framework that integrates Monte Carlo Tree Search (MCTS) with dynamic key-value retrieval to enable iterative, context-aware reasoning.
Outcome:	The proposed framework integrates Monte Carlo Tree Search (MCTS) with dynamic key-value (KV) retrieval to enable iterative, context-aware reasoning.

Copied to clipboard

Chenhui Mao, Yuanting Lei, Zhixiang Wei, Ming Liang, Zhixiang Wang, Jingxuan Xu, Dajun Chen, Wei Jiang, Yong Li

Training Language Model to Critique for Better Refinement (2025.findings-acl)

Challenge:	Entropy-Guided Stepwise Scaling (EGSS) is a novel TTS framework for software engineering tasks.
Approach:	They propose an entropy-guided stepwise scaling framework that balances efficiency and effectiveness through entropic-guide encoding and robust test-suite augmentation.
Outcome:	EGSS boosts performance by 5–10% across all evaluated models, and reduces inference-time token usage by over 28% . compared to existing methods, EGS reduces token usage and reduce inference time by over 20% .

Copied to clipboard

Tianshu Yu, Chao Xiang, Mingchuan Yang, Pei Ke, Bosi Wen, Cunxiang Wang, Jiale Cheng, Li Zhang, Xinyu Mu, Chuxiong Sun, Minlie Huang

CollabCoder: Plan-Code Co-Evolution via Collaborative Decision-Making for Efficient Code Generation (2026.findings-acl)

Challenge:	Large language models (LLMs) have remarkable evaluation and critique capabilities, providing insightful feedback and identifying flaws in various tasks.
Approach:	They propose a framework to train critic models using refinement signals to generate feedback loops where critiques guide the model in refining its responses.
Outcome:	The proposed framework outperforms traditional methods and open-source models in terms of critique quality and refinement outcomes.

Copied to clipboard

Duy Tung Doan, Quang Huy Phung, Dzung Nguyen, Khac-Hoai Nam Bui

Programming by Example meets Historical Linguistics: A Large Language Model Based Approach to Sound Law Induction (2025.acl-long)

Challenge:	Existing multi-agent code generation frameworks are constrained by static planning, isolated execution, high computational overhead, and limited adaptability to complex tasks.
Approach:	They propose a plan-code co-evolution framework that allows dynamic multi-agent collaboration to improve code quality and robustness across tasks.
Outcome:	The proposed framework improves code quality and robustness across tasks while reducing the number of API calls by an average of 4-10 per execution.

Copied to clipboard

Atharva Naik, Darsh Agrawal, Hong Sng, Clayton Marr, Kexun Zhang, Nathaniel Romney Robinson, Kalvin Chang, Rebecca Byrnes, Aravind Mysore, Carolyn Rose, David R. Mortensen

Exploring Coding Spot: Understanding Parametric Contributions to LLM Coding Performance (2026.findings-acl)

Challenge:	Historical linguists have written programs that convert reconstructed words into their attested descendants via ordered string rewrite functions.
Approach:	They propose to use a model to generate a "similar distribution" for sound law induction . they propose four kinds of methods with varying amounts of inductive bias to investigate best performance .
Outcome:	The proposed model shows that it can be fine tuned with training data and evaluation data.

Copied to clipboard

Dongjun Kim, Minhyuk Kim, Yongchan Chun, Chanjun Park, Heuiseok Lim

Subtle Errors in Reasoning: Preference Learning via Error-injected Self-editing (2025.acl-long)

Challenge:	Large Language Models (LLMs) have demonstrated proficiency in code generation and comprehension across multiple programming languages.
Approach:	They propose a parameter-localized subset of LLMs that facilitates coding capabilities.
Outcome:	The proposed model significantly improves performance on coding tasks while preserving non-coding functionalities.

Copied to clipboard

Kaishuai Xu, Tiezheng Yu, Wenjun Hou, Yi Cheng, Chak Tou Leong, Liangyou Li, Xin Jiang, Lifeng Shang, Qun Liu, Wenjie Li

RedCoder: Automated Multi-Turn Red Teaming for Code LLMs (2026.acl-long)

Challenge:	Existing studies to improve mathematical ability typically involve applying preference learning to step-wise solution pairs, but they overlook critical subtle errors.
Approach:	They propose a preference learning framework that injects predefined subtle errors into pivotal tokens to construct hard pairs for error mitigation.
Outcome:	Extensive experiments show that the proposed framework improves on Qwen2-7B-Instruct and MATH with 4.5K training samples.

Copied to clipboard

Wenjie Jacky Mo, Qin Liu, Xiaofei Wen, Dongwon Jung, Hadi Askari, Wenxuan Zhou, Zhe Zhao, Muhao Chen

Split-Merge: Scalable and Memory-Efficient Merging of Expert LLMs (2025.emnlp-main)

Challenge:	Existing red-teaming approaches for code generation rely on extensive human effort and are prone to generating malicious code under adversarial environments.
Approach:	They propose a red-teaming agent that engages victim models in multi-turn conversations to elicit vulnerable code.
Outcome:	Experiments show that RedCoder outperforms red-teaming methods in inducing vulnerabilities in code generation.

Copied to clipboard

Sruthi Gorantla, Aditya Rawal, Devamanyu Hazarika, Kaixiang Lin, Mingyi Hong, Mahdi Namazifar

MARS2: Scaling Multi-Agent Tree Search via Reinforcement Learning for Code Generation (2026.acl-long)

Challenge:	a zero-shot merging framework for large language models consolidates specialized domain experts into a single model without any further training.
Approach:	They propose a zero-shot merging framework that consolidates specialized domain experts into a single model without further training.
Outcome:	Experiments on code generation, mathematical reasoning, medical question answering, and instruction-following benchmarks confirm the versatility and effectiveness of the proposed framework.

Copied to clipboard

Pengfei Li, Shijie Wang, Fangyuan Li, Yikun Fu, Kaifeng Liu, Kaiyan Zhang, Dazhi Zhang, Yuqiang Li, Biqing Qi, Bowen Zhou

CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges (2026.findings-acl)

Challenge:	Existing approaches to reinforcement learning are decoupled from structured search due to limited trajectory diversity.
Approach:	They propose a unified RL framework that integrates multiple agents within a shared tree-structured search environment.
Outcome:	Experiments show that MARS2 improves performance across diverse model combinations and training settings.

Copied to clipboard

Zihan Wang, Lam Nguyen, Zhengyang Zhao, Mengyue Yang, Chengwei Qin, Yujiu Yang, Linyi Yang

Challenge:	Increasing saturation of web data limits further scaling of model intelligence.
Approach:	They propose a benchmark to evaluate machine creativity in code generation that combines combinatorial and exploratory creativity through reverse engineering and self-play.
Outcome:	The proposed benchmark targets combinatorial and exploratory creativity through reverse engineering and self-play.

Squeezed Attention: Accelerating Long Context Length LLM Inference (2025.acl-long)

Copied to clipboard

Coleman Richard Charles Hooper, Sehoon Kim, Hiva Mohammadzadeh, Monishwaran Maheswaran, Sebastian Zhao, June Paik, Michael W. Mahoney, Kurt Keutzer, Amir Gholami

A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code (2026.findings-acl)

Challenge:	Emerging Large Language Models require long input context to perform complex tasks.
Approach:	They propose an algorithm to reduce the complexity of attention with respect to the fixed context length.
Outcome:	The proposed method reduces the complexity of attention from linear to logarithmic with respect to the fixed context length.

Copied to clipboard

Keke Lian, Wang Bin, Lei Zhang, Libo Chen, Junjie Wang, Ziming Zhao, Yujiu Yang, Miaoqian Lin, Haotong Duan, Haoran Zhao, Shuang Liao, Mingda Guo, Quan Jiazheng, Yilu Zhong, Chenhao He, Chen Zichuan, Jie Wu, Haoling Li, Zhaoxuan Li, Jiongchi Yu, Hui LI, Dong Zhang

ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities (2025.acl-long)

Challenge:	Existing security evaluation benchmarks lack relevance to real-world AI programming tasks . current LLMs struggle with secure coding, research shows .
Approach:	They propose a repository-level evaluation benchmark to assess security of AI-generated code.
Outcome:	The proposed framework mirrors real-world AI programming tasks and offers valuable insights into the state of AI code generation.

Copied to clipboard

Zhaochen Hong, Haofei Yu, Jiaxuan You

RouterHGC: Optimized Router for LLM-based Multi-Agent Systems via Heterogeneous Graph Contrastive Learning (2026.findings-acl)

Challenge:	Traditional self-consistency methods fail to capture subtle semantic errors in multi-step tasks.
Approach:	They propose a tree-based evaluation framework that measures LLMs’ ability to preserve semantic consistency during reversible transformations.
Outcome:	The proposed framework measures generalization abilities across models from 1.5B to 72B and can be used to benchmark LLMs without constructing new datasets.

Copied to clipboard

Yitao Xiao, Shaoyong Guo, Guoming Yang, Qingnan Wang, Yinlin Ren, Xuesong Qiu, Qi Feng

OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models (2025.acl-long)

Challenge:	Large Language Models (LLMs)-driven Multi-Agent Systems (MAS) have demonstrated remarkable scalability and generalizability across complex tasks.
Approach:	They propose a new framework for routing using large language models . they formalize routing as node selection through edge-weight prediction .
Outcome:	The proposed framework outperforms the best single LLM and baselines on five datasets . it achieves 0.80%–6.17% accuracy gains on MATH and HotpotQA while reducing inference cost by 27.40%.

Copied to clipboard

Siming Huang, Tianhao Cheng, Jason Klein Liu, Weidi Xu, Jiaran Hao, Liuyihan Song, Yang Xu, Jian Yang, Jiaheng Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Xianzhen Luo, Qiufeng Wang, YuanTao Fan, Qingfu Zhu, Zhaoxiang Zhang, Yang Gao, Jie Fu, Qian Liu, Houyi Li, Ge Zhang, Yuan Qi, Xu Yinghui, Wei Chu, Zili Wang

AutoVecCoder: Teaching LLMs to Generate Explicitly Vectorized Code (2026.findings-acl)

Challenge:	Code LLMs lack reproducible data pipelines and training protocols for reproducible advancements in code intelligence.
Approach:	They propose a top-tier code LLM that releases model weights and inference code . reproducible data pipelines, rigorous experimental ablation results and training protocols are included .
Outcome:	The proposed model achieves comparable performance to leading models and serves as an "open cookbook" reproducible training data, rigorous experimental ablation results, and detailed training protocols are also included in the model.

Copied to clipboard

ShangZhan Li, Xinyu Yin, Xuanyu Jin, Ye He, Yuxin Zhou, Yuxuan Li, Xu Han, Wanxiang Che, Qi Shi, Ting Liu, Maosong Sun

Reasoning While Asking: Transforming Reasoning Large Language Models from Passive Solvers to Proactive Inquirers (2026.acl-long)

Challenge:	Current development practices face a dichotomy between automation and performance.
Approach:	They propose a framework to empower LLMs with the capability of automated explicit vectorization.
Outcome:	The proposed framework achieves state-of-the-art performance on the SSE and AVX subsets of SimdBench.

Copied to clipboard

Xin Chen, Feng Jiang, Yiqian Zhang, Hardy Chen, Shuo Yan, Wenya Xie, Min Yang, Shujian Huang

IntentCoding: Amplifying User Intent in Code Generation (2026.findings-acl)

Challenge:	Existing reasoning-oriented LLMs lack a blind self-thinking paradigm . current models fail to recognize when their reasoning is underinformed or based on ambiguous user instructions .
Approach:	They propose a new reasoning paradigm that transforms LLMs into proactive inquirers that interleave reasoning with clarification.
Outcome:	The proposed model outperforms baseline models on mathematical reasoning, code generation, and document editing.

Copied to clipboard

Zheng Fang, Yihong Dong, Lili Mou, Dongming Jin, Zhi Jin, Ge Li

Castle: Causal Cascade Updates in Relational Databases with Large Language Models (2025.emnlp-main)

Challenge:	IntentCoding captures the influence of user intent by masking out the intent, and integrates seamlessly with existing decoding procedures.
Approach:	They propose a decoding strategy that captures the influence of user intent by masking out the intent and applies a multi-strength ensemble mechanism to amplify the effect of user intention during generation.
Outcome:	The proposed model significantly improves both constraint satisfaction and functional correctness compared to greedy decoding approaches.

Copied to clipboard

Yongye Su, Yucheng Zhang, Zeru Shi, Bruno Ribeiro, Elisa Bertino

SSG: Logit-Balanced Vocabulary Partitioning for LLM Watermarking (2026.acl-long)

Challenge:	Existing approaches focus primarily on SELECT queries, neglecting the challenges of SQL update operations and their ripple effects.
Approach:	They introduce the first framework for schema-only cascade update generation using large language models.
Outcome:	The framework can produce accurate SQL updates without revealing table content . it can also determine which columns must be updated and how updates propagate through the schema, causing cascading updates .

Copied to clipboard

Chenxi Gu, Xiaoning Du, John C. Grundy

PaT: Planning-after-Trial for Efficient Test-Time Code Generation (2026.acl-long)

Challenge:	Large language models can generate highquality, human-like content, but they also pose risks such as infringement of proprietary interests, misuse of outputs, and spread harmful misinformation.
Approach:	They propose a method that partitions the vocabulary into two logit-balanced subsets and lifts the lower bound of watermark strength for each token prediction.
Outcome:	The proposed method lifts the lower bound of watermark strength for each token prediction, thereby improving watermark detectability.

Copied to clipboard

Youngsik Yoon, Sungjae Lee, Seockbean Song, Siwei Wang, Wei Chen, Jungseul Ok

Ro-SLM: Onboard Small Language Models for Robot Task Planning and Operation Code Generation (2026.findings-acl)

Challenge:	Existing methods for scaling test-time computation are rigid and inefficient . a heterogeneous configuration achieves performance comparable to a large homogeneously model .
Approach:	They propose an adaptive planning policy that invokes a planner only upon verification failure.
Outcome:	The proposed model achieves comparable performance to a large homogeneous model while reducing inference cost by approximately 69% across multiple benchmarks and model families.

Copied to clipboard

Wenhao Wang, Yanyan Li, Long Jiao, Jiawei Yuan

AutoSUIT Bench - Automated Security UnIt Test Benchmark for LLM Coding (2026.findings-acl)

Challenge:	Recent advances in large language models (LLMs) provide robots with contextual reasoning abilities to comprehend human instructions.
Approach:	They propose a framework that enables reliable SLM-driven robot operation by distilling LLMs’ knowledge and reasoning.
Outcome:	The proposed framework enables reliable SLM-driven robot operation by distilling LLMs’ knowledge and reasoning.

Copied to clipboard

Samuel Osebe, Fan Yang, Junyi Li, Yue Gu, Yongxin Wang, Satyapriya Krishna, Kai-Wei Chang, Aram Galstyan, Rahul Gupta, Weitong Ruan

Doc2Chart: Intent-Driven Zero-Shot Chart Generation from Documents (2025.emnlp-main)

Challenge:	Large Language Models (LLMs) are evolving rapidly on code generation tasks.
Approach:	They propose to automate the vulnerability code benchmark creation with iterative auto validation.
Outcome:	The proposed benchmark covers 232 CWE categories across C/C++, Java, and Python languages.

Copied to clipboard

Akriti Jain, Pritika Ramu, Aparna Garimella, Apoorv Saxena

RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories (2026.findings-acl)

Challenge:	Large Language Models have demonstrated strong capabilities in transforming text descriptions or tables to data visualizations . however, it is not straightforward to apply these methods directly for a more real-world use case of visualizing data from long documents .
Approach:	They propose an unsupervised method for generating intent-based charts from documents . they propose an attribution-based metric that uses a structured textual representation of charts .
Outcome:	The proposed method outperforms baselines in terms of chart data accuracy and chart type over baselines.

Copied to clipboard

Yanlin Wang, Ziyao Zhang, Chong Wang, Xinyi Xu, Mingwei Liu, Yong Wang, Jiachi Chen, Zibin Zheng

Towards Trustworthy Smart Contract Synthesis: A Multi-Agent Framework with Lean-Based Verification (2026.acl-long)

Challenge:	Existing benchmarks for large language models fail to capture complex interplay between functionality and security.
Approach:	They propose a benchmark for secure code generation constructed from real-world, high-risk Java repositories.
Outcome:	The proposed benchmarks highlight the gap between functional and secure code generation in LLMs.

Copied to clipboard

Bowei Zhang, Hanbing Liu, Qixin Tian, Siyu Chen, Ziyuan Wang, Qi Qi

CodeRise: Bootstrapping LLMs for Ultra Low-Resource Programming Languages via Progressive Self-Refinement Curriculum (2026.findings-acl)

Challenge:	Smart Contracts are the foundation of Decentralized Finance (DeFi), executing financial logic without trusted intermediaries.
Approach:	They propose a framework that integrates LLM-based generation with Lean-based auto-formalization and verification.
Outcome:	LeVer is the first trustworthy smart contract synthesis framework that integrates LLM-based generation with Lean-based auto-formalization and verification.

Copied to clipboard

Tengfei Wen, Xuanang Chen, Ben He, Xiaoliang Cong, Le Sun

RealChart2Code: Bridging the Gap in Real-World Chart-to-Code Generation via Multi-Task Evaluation (2026.acl-long)

Challenge:	Existing methods for training data generation for low-resource languages suffer from a cold-start problem and lack diversity.
Approach:	They propose a two-stage framework that generates a high-quality, diverse, and progressively complex curriculum for Ultra Low-Resource Programming Languages (ULRPLs) they leverage the full formal syntax of the target language as structural guidance and apply a biased sampling strategy over library modules.
Outcome:	The proposed framework outperforms training-free and training-based baselines on two ULRPLs, Tengo and Janet.

Copied to clipboard

Jiajun Zhang, Yuying Li, Zhixun Li, Xingyu Guo, Jingzhuo Wu, Leqi Zheng, Yiran Yang, Jianke Zhang, Qingbin Li, Shannan Yan, Changguo Jia, Junfei Wu, Zilei Wang, Qiang Liu, Liang Wang

SQLAgent: Learning to Explore Before Generating as a Data Engineer (2026.findings-acl)

Challenge:	Vision-Language Models (VLMs) have demonstrated impressive capabilities in code generation across various domains, but their ability to replicate complex, multi-panel visualizations remains largely unassessed.
Approach:	They propose a large-scale benchmark to evaluate chart generation from large- scale raw data and assess iterative code refinement in a multi-turn conversational setting.
Outcome:	The new benchmark evaluates 14 leading VLMs on real-world data and shows they struggle with complex plot structures and authentic data.

Copied to clipboard

Wenjia Jiang, Yiwei Wang, Boyan Han, Joey Tianyi Zhou, Chi Zhang

ReCode: Reinforcing Code Generation with Reasoning-Process Rewards (2026.acl-long)

Challenge:	Existing approaches to large language models fail to generalize in complex, real-world settings due to database-specific nature of SQL reasoning.
Approach:	They propose a two-stage LLM-based framework that decouples knowledge acquisition from query generation.
Outcome:	The proposed framework significantly improves accuracy over baselines on large-scale benchmarks.

Copied to clipboard

Lishui Fan, Yu Zhang, Mouxiang Chen, Zhongxin Liu

Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Quality (2026.findings-acl)

Challenge:	Bringing process-level supervision into RL often neglects optimizing reasoning quality.
Approach:	They propose a framework for RL that integrates reasoning-process rewards with strict execution outcomes and a benchmark comprising preference pairs of superior and inferior reasoning processes.
Outcome:	The proposed framework outperforms the base version of ReCode by 16.1% and reaches performance comparable to GPT-4-Turbo.

Copied to clipboard

Qipeng Xie, Zi Liang, Jiafei Wu, Yufei Chen, Weizheng Wang, Wenao Ma, Zhong Ming, Haiqin Yang, Kaishun Wu

Tandem: Riding Together with Large and Small Language Models for Efficient Reasoning (2026.findings-acl)

Challenge:	Existing studies on prompt engineering have focused on optimizing models for performance under stylistic perturbations.
Approach:	They conduct the first analysis of n-gram token-level mechanisms . they find that higher average performance is inherently associated with lower variance and greater stability.
Outcome:	The proposed model reduces the variance of the generated code by 40% . the proposed model is based on a large-scale dataset of 132,000 prompt variants .

Copied to clipboard

Zichuan Fu, Xian Wu, Guojing Li, Yejing Wang, Yijun Chen, Zhao Zihao, Luo Yixuan, Hanyu Yan, Yefeng Zheng, Xiangyu Zhao

Rose-SQL: Role-State Evolution Guided Structured Reasoning for Multi-Turn Text-to-SQL (2026.acl-long)

Challenge:	Recent advances in large language models (LLMs) have catalyzed the rise of reasoningintensive inference paradigms, where models perform explicit step-by-step reasoning before generating final answers.
Approach:	They propose a large-small LLM collaboration framework that synergizes large and small language models to achieve high-quality reasoning with significantly reduced computational cost.
Outcome:	The proposed framework outperforms the mentor LLM while preserving the benefits of the thinking paradigm of LLMs.

Copied to clipboard

Le Zhou, Feng Yao, Fengcai Qiao, Bo Xu, Fangyuan Wang, Boyan Xu

Reinforcement Learning for Diffusion LLMs via Energy-Based Gibbs Alignment (2026.acl-long)

Challenge:	Existing approaches to multi-turn Text-to-SQL tasks rely on unstable APIs or expensive fine-tuning.
Approach:	They propose a training-free framework that leverages small-scale LRMs through in-context learning to enable accurate context-dependent parsing.
Outcome:	The proposed framework outperforms in-context learning baselines at the 4B scale and surpasses state-of-the-art models at the 8B and 14B scales.

Copied to clipboard

Yijia Fan, Jing Yang, Mingyu Liu, Kaitong Cai, Jian Wang, Keze Wang, Jusheng Zhang

DuET: Dual Execution for Test Output Prediction with Generated Code and Pseudocode (2026.findings-acl)

Challenge:	Diffusion Large Language Models (dLLMs) offer parallel decoding and bidirectional context modeling . aligning dLLms with reinforcement learning (RL) remains a challenge .
Approach:	They propose a variational framework that reformulates RL for dLLMs as a distribution matching problem.
Outcome:	The proposed framework reformulates RL for dLLMs as a distribution matching problem.

Copied to clipboard

Hojae Han, Jaejin Kim, Seung-won Hwang, Yu Jin Kim, Moontae Lee

CodeRM-NT: Reward Model for Code RL without Unit Tests (2026.findings-acl)

Challenge:	Recent studies have shown that test output prediction is difficult to achieve due to code errors.
Approach:	They propose a framework that grounds prediction on error-resilient pseudocode and simulates execution via LLM reasoning to overcome limitations of direct execution suffering from code errors.
Outcome:	The proposed framework improves Pass@1 on LiveCodeBench, BigCodeBech-Hard, DevEval and HumanEval(+) and improves on pass@1 by 13.6 pp.

Copied to clipboard

Xiao Xia, Dan Zhang, Tianrui Sun

GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities (2026.acl-long)

Challenge:	Existing methods rely on unit tests to evaluate code correctness and provide rewards, but these methods are difficult to verify at scale.
Approach:	They propose a code reward model that leverages Monte Carlo Tree Search guided by LLMs to generate code snippets and judges execution traces to annotate code with reward signals.
Outcome:	The proposed model outperforms synthetic unit tests on multiple code generation benchmarks and improves curriculum learning.

Copied to clipboard

Diganta Misra, Nizar Islah, Victor May, Brice Rauby, Zihan Wang, Justine Gehring, Antonio Orvieto, Muawiz Sajjad Chaudhary, Eilif B. Muller, Irina Rish, Samira Ebrahimi Kahou, Massimo Caccia